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ABSTRACT 


y 

This  thesis  is  concerned  with  the  control  of  entry  to  queueing 
systems.  An  M/M/1  model  with  a  single  class  of  customers  and  an 
infinite  time  horizon  studied  by  Naor  (1969)  provides  the  starting 
point  for  this  work.  Each  customer  receives  a  fixed  reward  for 
service  and  pays  a  holding  cost  at  a  fixed  rate  per  unit  of  time  he 
spends  in  the  system.  Each  customer  may  choose  to  join  the  system 
or  not.  A  self-optimizing  customer  decides  whether  or  not  to  join 
by  acting  to  maximize  his  own  expected  net  benefit.  A  social 
optimizing  customer  decides  by  acting  to  maximize  the  gain  rate  of 
the  system,  the  sum  of  the  expected  net  benefits  per  unit  time  of  all 
arrivals.  Socially  optimal  control  of  the  model  is  provided  by 
establishing  a  balking  point  for  the  customers  that  is  no  greater 
than  the  balking  point  a  self -optimizing  customer  would  determine. 
Naor's  approach  is  compared  with  the  semi-Markov  decision  process 
formulation  of  Yechlali  (1971).  The  results  for  this  first  model 
are  extended  to  show  that  the  gain  rate  can  only  increase  as  the 
arrival  rate  of  the  customers  increases  while  the  social  balking 
point  can  only  decrease  as  the  arrival  rate  increases. 

A  semi-Markov  approach  is  used  to  formulate  an  expanded  model 
with  several  classes  of  customers,  each  with  its  own  reward  and 
holding  cost  rate.  Socially  optimal  control  of  this  model  is  shown 
to  be  provided  by  establishing  a  balking  point  for  each  class  that 
is  no  greater  than  the  balking  point  a  member  of  the  class  would 
determine  if  he  acted  to  maximize  his  own  expected  net  benefit. 


A  semi-Markov  approach  is  also  used  to  extend  the  several  class 
model  to  include  Erlang  service  times.  A  heuristic  solution  technique 
based  on  policy  iteration  and  a  solution  technique  using  mixed  integer 
programming  are  presented. 

The  several  class  models  are  applied  to  the  problem  of  deter¬ 
mining  an  optimal  policy  for  controlling  the  entry  of  commercial 
aircraft  to  the  landing  queue  at  the  Greater  Pittsburgh  International 
Airport.  A  socially  optimal  control  policy  is  found  and  analyzed. 

Finally,  the  semi-Markov  approach  is  used  to  formulate  three 
other  models: 

1)  A  nonpreemptive  priority  model. 

2)  A  class  dependent  service  rate  model. 

3)  A  nonpreemptive  priority  model  with  class  dependent 
service  rates. 
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CHAPTER  I 


INTRODUCTION 

Until  quite  recently,  queueing  theory  has  been  used  primarily 
as  a  descriptive  tool.  In  this  mode,  an  existing  or  proposed  queueing 
system  is  "allowed  to  operate"  and  its  behavior  is  described  by  one  or 
more  measures  such  as  expec  ad  waiting  time  and  expected  queue  length. 
Currently,  an  increasing  portion  of  the  queueing  literature  is  being 
devoted  to  the  use  of  queueing  theory  to  control  ard  optimize  the 
operation  of  a  system.  Here,  queueing  analyses  are  used  to  design  a 
system  or  to  develop  an  operating  policy  to  control  its  operation  so 
that  certain  standards  are  met.  For  Instance,  a  policy  might  be  sought 
to  keep  operating  costs  below  some  upper  bound  while  maximizing 
throughput  of  the  queue.  This  more  recent  approach  of  designing  or 
controlling  a  queueing  system  based  on  an  optimum  operating  policy  is 
adopted  throughout  this  work.  The  policies  sought  are  those  which 
maximize  gain  per  unit  time.^ 

1.1  Problem  Statement 

Suppose  that  customers  from  M  homogeneous  classes  arrive  at  a 
service  facility  with  a  single  server.  Th  classes  are  homogeneous  in 
in  the  sense  that  each  customer  in  a  particular  class,  m,  faces  the 
same  cost  structure  at  the  service  facility,  namely,  a  reward  for 

Hftiile  gain  can  be  given  a  broad  range  of  interpretation,  it  is 
perhaps  easiest  to  think  of  gain  as  financial  gain,  although  the  term 
is  not  restricted  to  dollars. 
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service  and  a  cost  per  unit  tine  the  customer  spends  in  the  system. 
Suppose  also  that  each  customer  may  either  join  the  system  or  not. 

Many  economic  viewpoints  or  objectives  can  be  used  to  make  the  join  or 
balk  decisions.  Two  viewpoints  of  interest  are  that  of  an  individual 
customer  and  that  of  the  group  of  customers  or  society  acting  as  a 
whole . 


The  first  viewpoint  gives  rise  to  the  individual  optimum 

problem  in  which  each  arriving  customer  makes  his  own  decision  whether 

or  not  to  join  the  system  so  that  his  own  expected  net  benefit  is 

maximized.  Arriving  customers  are  assumed  to  be  able  to  determine  the 

state  of  the  system,  e.g.,  the  number  of  customers  in  the  system.  A 

customer's  expected  net  benefit  for  joining  when  i  customers  are  ia 

the  system  is  his  reward  R  minus  C  times  his  expected  time  in  the 

mm 

system,  given  that  i  customers  are  in  front  of  him.  Unless  otherwise 
stated,  the  service  discipline  used  in  the  models  is  first  come,  first 
served.  A  customer's  net  benefit  for  balking  is  zero. 

The  other  viewpoint  produces  the  social  optimum  problem  where 
the  decision  of  whether  or  not  to  allow  an  arrival  to  join  the  system 
is  made  by  a  single  decision  maker  who  seeks  to  maximize  the  sum  of  the 
expected  net  benefits  per  unit  time  of  all  arriving  customers.  One  way 
to  view  this  decision  anker  i3  to  think  of  him  as  an  administrator  who 
collects  all  rewards  and  pays  all  the  costs  for  the  customers  who  use 
the  system,  and  then  distributes  the  gains  among  all  arrivals  whether 


or  not  they  joined  the  system.  While  solution  of  the  social  optimum 
problem  is  the  major  goal,  the  solution  of  the  individual  optimum 
problem  is  also  obtained  and  used  to  limit  the  search  for  the  solution 
to  the  social  optimum  problem. 
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The  solution  to  the  social  optimum  problem  is  an  optimum  policy 

(i.e.,  a  join  or  balk  decision  for  each  class  of  customers,  for  each 

possible  state  of  the  system) .  Several  assumptions  about  the  arrival 

distribution  of  the  customers  and  the  service  time  distribution  of  the 

server  are  required  to  solve  the  problem  in  a  straightforward  manner. 

These  assumptions  are  noted  in  the  outline  of  the  dissertation  and  are 

explained  more  fully  in  the  various  chapters. 

For  several  of  the  models  presented  in  this  paper,  optimal 

control  of  the  entry  of  customers  into  the  system  is  shown  to  be 

provided  by  a  set  of  balking  points  n  ■  (n^,  n^,  ...»  n^)  ,  where 

class  m  joins  if  the  state  of  the  system  is  less  than  n  .  Tor  all  the 

m 

models  presented  in  this  paper,  the  balking  point  for  a  self-optimizing 
customer  of  class  m  is  shown  to  be  at  least  as  large  as  that  of 
customers  of  class  m  acting  in  a  socially  optimal  manner.  Thus, 
compared  with  socially  optimal  behavior,  self -optimizing  customers 
tend  to  overcongest  the  system. 

The  particular  application  to  which  the  results  obtained  will  be 

applied  is  the  landing  queue  of  an  airport.  It  should  be  noted, 

however,  that  applications  as  diverse  as  deciding  on  the  number  of 

skiers  to  allow  at  a  ski  resort  on  a  holiday  and  determining  the 

number  of  terminals  allowed  to  tie  into  a  computer  system  can  be  molded 

into  queueing  system  entry  control  problems.  In  the  landing  queue  at 

an  airport,  various  types  of  commercial  aircraft  are  the  classes  of 

customers  in  the  landing  queue  system.  For  each  class,  R  is  a  measure 

m 

of  the  gain  when  a  class  m  aircraft  lands.  C  is  the  cost  per  unit 
time  of  keeping  an  aircraft  of  class  a  in  the  landing  queue,  including 
such  costs  as  crew  wages  and  fuel. 

> 

* 

a 

i 
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Entry  of  aircraft  into  an  airport's  landing  queue  Is  currently 
being  controlled  in  several  manners  for  safety  reasons.  At  all  major 
airports,  the  air  traffic  controllers  maintain  an  upper  bound  on  the 
size  of  the  landing  queue.  If  the  queue  is  full,  arriving  aircraft 
are  denied  entry  into  the  landing  queue  and  are  kept  under  the  command 
of  controllers  in  another  air  traffic  sector.  Also,  at  certain  air¬ 
ports  designated  as  high  density  traffic  airports,  the  Federal  Aviation 
Administration  (FAA)  limits  the  number  of  Instrument  flight  rule  (IFR) 
operations  (take  offs  and  landings)  per  hour  allowed  during  peak 
traffic  periods.  These  limits  which  are  set  by  negotiation  within  the 
Industry  are  given  in  Table  1.1.  It  is  not  necessary  that  aircraft  in 
flight  which  are  not  admitted  to  the  landing  queue  be  forced  to  land 
elsewhere.  In  practice,  they  could  wait  if  they  choose.  The  model, 
however,  is  useful  in  determining  the  capacity  of  an  airport  landing 
queue  from  an  economic  viewpoint.  Thus,  the  entry  of  aircraft  into  the 
landing  queue  is  controlled  in  the  model  for  the  purposes  of  defining 
an  economical  workload  for  the  airport  or,  more  appropriately,  an 
economical  airport  for  the  workload.  The  former  can  be  accomplished 
by  schedule  modifications,  the  latter  by  airport  design. 

TABLE  1.1 

FAA  Limits  for  High  Density  Traffic  Airports 
IFR  Operations  Per  Hour  [FAA  (1974)] 


Class  of 

User 

JFK 

La  Guard ia 

Airport 

Newark 

O’ Hare 

Wash. 

National 

Air  Carrier 

70 

48 

40 

115 

40 

Air  Taxi 

5 

6 

10 

10 

8 

Other 

5 

6 

10 

10 

12 

5 


The  models  developed  in  the  first  few  chapters  of  this  paper  will 
be  used  to  determine  the  capacity  of  the  system  (from  an  economic 
viewpoint).  Since  the  aircraft  are  divided  into  several  classes,  it 
is  possible  that  the  capacity  of  the  system  may  vary  with  the  class  of 
an  arrival  since,  for  certain  states  of  the  system,  the  administrator 
may  admit  members  of  some  classes  but  not  of  others.  When  a  decision 
concerning  the  direction  the  development  of  a  model  is  to  take  is  made, 
the  appropriateness  of  the  direction  to  the  landing  queue  application 
is  the  criterion  on  which  the  choice  is  based . 

1.2  Outline  of  the  Dissertation 

Following  this  chapter.  Chapter  II  presents  a  literature  survey 
of  selected  works  in  the  area  of  control  of  entry  to  queues.  A  few 
works  on  the  control  of  the  server  ip  a  queueing  system  and  other 
general  topics  that  serve  as  background  for  this  work  are  also  included. 
Two  of  these  topics  are  air  traffic  control  and  the  estimation  of  the 
parameters  of  a  probability  distribution  function.  As  indicated  in  the 
flow  diagram  for  this  dissertation.  Figure  1.1,  the  flow  of  the 
dissertation  is  not  interrupted  by  skipping  Chapter  II. 

Model  I,  the  simplest  model  used  in  this  study,  is  presented  in 
Chapter  III.  A  single  class  of  customers  whose  arrival  forms  a 
Poisson  stream  to  a  single  server  with  exponentially  distributed 
service  times  is  modeled.  Thus,  this  model  is  an  M/M/l/n  queueing 
system  where  the  capacity  of  the  system,  n,  is  the  decision  variable. 

The  capacity  of  the  system,  n,  is  also  referred  to  as  the  balking 
point  af  the  customers;  that  is,  if  n  (or  more)  customers  are  in  the 
system,  an  arrival  balks  at  the  opportunity  to  join. 
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Model  I  has  been  solved  previously  by  Naor  (1969)  and  Yechiali 
(1971)  for  the  Individual  and  social  optimum  values  of  n.  Naor 
developed  his  results  using  algebra  and  the  properties  of  the  model, 
while  Yechiali  developed  his  results  by  treating  the  problem  as  a 
semi-Markov  decision  process.  Chapter  III  compares  the  work  of  the 
two  authors  and  extends  it  somewhat.  A.  few  sidelights  such  as  tolls 
(which  both  authors  studied)  are  examined  as  one  means  of  implementing 
the  solution  to  the  social  optimum  problem.  The  toll  charged  to  a 
customer  who  joins  the  system  alters  his  net  benefit  so  that  self- 
optimizing  customers  find  it  desirable  to  join  the  system  only  when  a 
social  optimizing  administrator  would  have  let  them  join. 

In  Chapter  IV,  M  classes  of  customers  are  considered.  Each 

class  m  has  its  own  reward  for  service  R^,  cost  per  unit  time  in  the 

system  C  ,  and  mean  arrival  rate  X  ,  but  all  have  the  same  service 
m  m 

rate  u.  The  work  of  Naor  (1969)  and  Yechiali  (1971)  is  extended  to 
this  model.  In  particular,  the  form  of  the  optimal  policy  carries  over 
to  this  new  model.  Model  II.  The  decision  variable  in  this  model  is  a 
policy,  a  set  of  join  or  balk  decisions  for  each  class  of  customers, 
for  each  possible  state  of  the  system.  Recall  that  the  number  of 
customers  in  the  system  is  the  state  of  the  system.  The  balking  point 
of  Model  I  now  becomes  a  vector  n  of  balking  points  for  each  class. 

This  implies  that  some  classes  of  customers  may  receive  preferential 
treatment  in  the  operation  of  the  system.  1'olicy  iteration  and  linear 
programming  solution  techniques  are  presented  for  the  semi-Markov 
decision  process  formulation  of  the  problem. 

The  service  time  distribution  of  the  M  class  model  is  generalized 
from  an  exponential  to  an  Erlang  distribution  in  Chapter  V.  This  change 
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allows  more  flexibility  in  modeling  the  service  time  distribution  of  an 
actual  system.  Again,  the  model  is  structured  as  a  semi-Markov 
decision  process.  One  solution  method  developed,  based  on  policy 
iteration,  is  easy  to  implement  but  does  not  guarantee  an  optimal 
solution.  It  assumes  that  the  optimal  solution  is  again  a  vector  of 
balking  points  and  performs  a  limited  search  for  a  good  solution.  A 
second  method  permits  an  exhaustive  search  of  all  possible  policies 
through  a  mixed  integer  programming  formulation. 

As  indicated  in  Figure  1.1,  Chapters  III  through  V  cover  the 
three  major  models  developed  in  this  dissertation.  Model  II  and  the 
Erlang  model,  Model  III,  are  applied  to  a  sample  airport  landing  queue 
problem  in  Chapter  VI.  Airline  data,  FAA  data,  and  data  taken  at  the 
Greater  Pittsburgh  International  Airport  and  Washington  National 
Airport  are  used  in  Chapter  VI  to  estimate  the  parameters  required  for 
the  models.  The  models  are  then  used  to  determine  how  entry  to  the 
landing  queue  should  be  controlled  to  maximize  the  social  optimum 
objective  function.  The  sensitivity  of  the  results  to  the  input 
parameters  is  also  examined. 

In  Chapter  VII,  three  variations  are  developed  for  Model  II. 
First,  the  service  discipline  is  changed  from  first  come,  first  served 
to  a  nonpreemptive  priority  service  discipline.  The  second  variation 
expands  the  flexibility  of  the  model  by  allowing  different  mean  service 
rates  for  each  class  or',  customers.  The  last  variation  combines  the 
first  two. 

A  summary  of  the  entire  dissertation  and  the  conclusions  reached 
is  presented  in  Chapter  VIII  along  with  a  list  of  potential  uses  for 


- .  i'ii. 
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the  models  developed  in  the  earlier  chapters.  Suggested  areas  for 


future  research  are  also  presented. 

There  are  four  appendices  to  this  paper.  A  glossary  of  the 
notations  used  in  the  paper  is  presented  in  Appendix  A  to  facilitate 
reading  the  more  mathematical  sections  of  this  dissertation.  Appendix 
B  contains  proofs  of  some  of  the  work  of  Naor  (1969)  and  an  example 
problem  solved  using  his  work.  A  brief  introduction  to  semi-Markov 
decision  processes  and  policy  iteration  along  with  an  example  problem 


illustrating  the  use  of  policy  iteration  comprise  Appendix  C. 
Appendix  D  is  a  listing  and  user's  guide  for  a  computer  program  that 
solves  Models  I,  II,  and  III. 


CHAPTER  II 


LITERATURE  SURVEY 

This  chapter  is  a  survey  of  selected  works  that  influenced  the 
writing  of  this  paper.  The  works  are  divided  into  three  areas:  1) 
control  of  arrivals,  2)  control  of  server,  and  3)  general.  A  short 
summary  of  the  papers  in  each  group  is  presented  in. alphabetical 
order  by  author's  name. 

2.1  Control  of  Arrivals 

Balachandran  and  Schaefer  (1975)  examine  an  M/G/l  queueing  model 
with  several  classes  of  customers.  Each  class  has  its  own  reward  for 
service cost  per  unit  time  in  the  system,  mean  service  rate,  and  mean 
arrival  rate  which  the  class  adjusts  based  on  expected  waiting  time. 

The  individual  optimum  policy  allows  the  service  facility  to  be 
dominated  by  a  single  class.  The  social  optimum  also  admits  only  one 
class,  but  it  may  not  be  the  same  class  as  the  individual  optimum. 
Various  techniques  are  explored  for  diversifying  access  to  the  facility. 

Balachandran  and  Schaefer  (1976)  consider  an  M/M/1  queueing  model 
with  K  classes  of  customers  where  each  class  adjusts  its  own  arrival 
rate  based  on  average  waiting  time  in  the  queue.  Each  customer  of 
class  1  receives  a  reward  for  service  and  loses  v^  for  each  unit  of 
time  spend  in  the  system.  If  A°  is  the  optimal  aggregate  arrival  rate, 
the  average  waiting  time  is  W  ■  l/(y  -  A0)  .  The  service  facility  is 
used  only  by  those  classes  for  which  g^  ■  v^/ (y  -  A0)  .  The  authors 
Introduce  admission  prices  to  equalize  the  attractiveness  of  the 
facility  among  the  classes. 
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Baiachandran  and  Tilt  (undated)  consider  decision  making  in 
queueing  systems  with  customers  given  the  choice  of  1)  joining,  2) 
balking,  or  3)  choosing  a  priority  through  payment  made  on  arrival. 
Several  models  are  examined  from  the  standpoint  of  a  noncooperative 
n  person  game.  One  of  the  models  treated  yields  an  individual  optimum 
for  the  GX/M/s/n  queue  which  is  analogous  to  that  obtained  by  Yechiali 
(1972). 

Edelson  and  Hildebrand  (1973)  reexamine  the  work  of  Naor  (1969) 
and  extend  it  to  several  classes  of  customers.  However,  the  authors 
are  interested  in  the  relationship  between  the  social  optimum  and  the 
revenue  maximizer's  optimum.  They  first  examine  conditions  under  which 
the  two  are  the  same.  Then,  they  examine  a  generalized  model  which 
includes  several  classes  of  customers,  each  with  its  own  reward  and 
cost  per  unit  time  in  the  system.  They  give  a  computational  technique 
for  finding  the  expected  number  of  each  class  of  customers  in  the 
system.  In  addition,  they  show  that  for  the  generalized  rase,  the 
revenue  maximizer's  optimum  balking  point  is  not  necessarily  leas  than 
the  social  optimum  (which  it  was  for  Naor's  model). 

Emmons  (1972)  considers  an  M/M/s  queueing  system  with  the 
following  cost  structure: 

a)  A  fixed  running  cost  rate. 

b)  An  expected  revenue  per  customer,  r. 

c)  An  overtime  running  cost  rate  called  K. 

The  system  is  run  for  a  finite  length  of  time  with  no  customers 
admitted  after  closing.  A  policy  of  admitting  customers  as  a  function 
of  number  in  the  system  and  time  to  closing  is  sought  which  maximizes 
the  operator's  total  expected  profit.  The  optimal  policy  admits 
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customers  only  if  the  number  in  the  system,  i,  is  such  that 
f^(t)  ■  i  -  f2(t)  *  where  f^(t)  and  f2(t)  are  give::..  If  y  is  the 
service  rate  and  r  ■  K/(cy),  thenf^Ct)  ■  ®,  so  that  customers  are 
always  admitted  when  f^(t)  or  more  are  in  the  system.  This  rule  can 
be  transformed  into  an  optimal  rejection  time  rule.  For  r  <  K/(cy), 
f 2 ( t t  “and  the  rule  cannot  be  transformed  into  an  optimal  rejection 
time  rule. 

Harrison  (1975)  studies  an  M/G/l  queueing  system  with  K  classes 
of  customers.  Each  class  has  its  own  arrival  rate,  reward  for  service, 
holding  cost,  and  service  time  distribution  function.  All  arrivals  are 
allowed  to  join  the  queue,  but  an  administrator  decides  at  the  comple¬ 
tion  of  each  service  which  class,  if  any,  to  aduj  :  next  to  service. 

The  objective  of  the  administrator  is  to  maximize  the  sum  of  the  dis¬ 
counted  net  benefits  of  all  customers  in  the  system.  A  nonpreemptive 
priority  discipline  (that  may  ignore  several  classes)  is  shown  to  be 
optimal. 

Knudsen  (1972)  considers  an  M/M/s/n  queueing  system.  He 
generalizes  the  work  of  Naor  (1969)  to  s  servers  and  to  a  nonlinear 
waiting  cost  function.  The  author  shows  that  if  the  net  benefit  to 
each  arriving  customer  is  a  decreasing  concave  function  of  the  number 
in  the  system,  then  Naor's  results  hold;  that  is,  the  revenue  maximiz¬ 
er's  optimum  balking  point  is  less  than  or  equal  to  the  social  optimum 
which  is  in  turn  less  than  or  equal  to  the  individual  optimum.  The 
author  also  discusses  the  shadow  price  aspect  of  the  tolls  charged  to 
customers  to  get  them  to  act  in  a  socially  optimal  manner.  Finally, 
he  discusses  the  difficulties  of  pricing  resources  in  a  stochastic 
situation. 


Lippman  and  Stidham  (1977)  examine  a  queueing  system  with  an 
arrival  rate  X  and  a  service  rate  y^  that  is  a  nondecreasing,  concave, 
bounded  above  function  of  i,  the  number  in  the  system.  Typically,  the 
sequence  y^  might  arise  from  an  M/M/s  system  where  y^  ■  y  *  min(s,i)  . 
Associated  with  each  customer  accepted  is  a  reward  r  which  is  a  random 
variable.  An  accepted  customer  joins  the  queue  and  incurs  a  waiting 
cost  of  h  per  unit  time  until  he  departs.  The  system  may  either  be 
controlled  by  society  as  a  whole  or  by  the  individual  customers. 

Control  of  the  system  is  exercised  to  maximize  expected  discounted  net 
benefits  over  an  infinite  or  finite  horizon.  The  paper  compares 
individual  and  social  optima  and  shows  that  regardless  of  system  state, 
remaining  horizon,  or  discount  rate,  a  customer  left  to  his  own  devices 
will  enter  the  system  whenever  the  social  optimum  calls  for  him  >.0 
enter.  In  addition,  the  customer  may  enter  when  the  social  optimum 
calls  for  him  to  balk.  The  authors  claim  that  this  discrepancy  is 
caused  by  the  failure  of  an  individual  customer  to  consider  the  short¬ 
fall  in  benefits  to  later  arriving  customers  caused  by  his  joining  the 
queue.  The  paper  also  examines  the  behavior  o,"  socially  and  individ¬ 
ually  optimal  policies  and  returns  as  functions  of  i,  n,  and  the 
discount  rate.  Finally,  the  paper  discusses  tolls  that  should  be 
charged  to  get  individuals  to  act  in  a  socially  optimal  manner. 

Littlechild  (1974)  considers  an  M/M/1  queueing  system  with 
customers  that  have  the  same  cost  of  waiting  but  different  rewards  for 
service.  He  develops  a  toll  to  charge  all  customers  to  reduce  the 
arrival  rate  so  that  social  benefit  is  maximized.  In  other  words,  the 
reduction  in  waiting  cost  due  to  the  smaller  arrival  rate  more  than 
offsets  the  loss  in  benefits  of  service  to  the  customers  as  a  whole. 
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Miller  (1969)  examines  an  M/M/s/s  queueing  system  with  K 
classes  of  customers.  Each  class  has  its  own  reward  for  service  and 
arrival  rate,  but  all  classes  have  the  same  service  rate.  When  a 
cuaLomer  arrives,  a  decision  is  made  whether  or  not  to  serve  him.  If 
thr  decision  is  made  not  to  serve,  the  customer  departs;  thus,  there 
is  no  queue  and  no  preemption.  The  objective  of  the  paper  is  to  find 
the  admittance  policy  which  maximizes  the  average  value  of  rewards  over 
an  infinite  planning  horizon.  When  the  system  is  in  state  j,  j  servers 
are  free  and  a  policy  such  as  serve  customers  of  classes  {1,  2,  3}  may 
be  chosen.  The  problem  is  formulated  as  a  semi-Markov  process  and 
policy  iteration  is  used  to  solve  for  an  optimal  policy.  The  paper 
also  describes  two  heuristic  methods  for  examining  a  generalized 
problem  that  allows  each  class  to  have  its  own  general  service  time 
distribution. 

Mine  and  Ohno  (1971)  consider  an  M/G/l  queueing  system  in  which 

the  number  of  customers  in  the  system  is  unavailable  or  too  expensive 

to  maintain.  Arriving  customers  are  accepted  during  the  time  interval 

(t  ,  t'),  starting  time  to  rejection  time,  and  rejected  with  compensa- 
o 

tion  after  time  t'.  The  server  runs  at  a  cost  rate  r  during  normal 

o 

hours,  (tQ,  T) ,  where  T  is  closing  time  (note  that  tQ  ■  t'  -  T) .  The 
server  runs  at  increased  rate  r^  after  T  (overtime)  and  at  reduced 
rate  r^  if  he  becomes  idle  after  t'  (goes  home).  The  paper  finds  the 
rejection  time  that  minimizes  total  expected  cost. 

Naor  (1969)  examines  an  M/M/1  queueing  system  in  which  customers 
receive  a  reward  R  for  service  and  pay  a  cost  of  C  per  unit  time  in  the 
system.  The  arrival  rate  is  X  and  the  service  rate  is  y.  Each  arrival 
may  choose  either  to  join  the  system  or  not.  Three  types  of  objective 
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functions  are  considered.  In  the  first,  the  individual  optimum 

problem,  each  customer  seeks  to  maximize  his  own  expected  net  gain. 

It  is  optimal  to  balk  when  ng  customers  are  in  the  system,  where 

n  is  the  greatest  integer  in  (Ry/C) .  This  is  simply  an  M/M/l/n 

system.  The  second  optimization  considers  the  collective  good  of  all 

arrivals  given  by  the  sum  of  their  expected  net  benefits  per  unit  time. 

n  a  n  is  now  the  maximum  number  allowed  in  the  system.  Reduction  to 
os 

n^  can  be  done  either  by  administrative  rule  or  by  levying  a  toll  on 
entering  cuatomers  to  reduce  nfl  to  nQ.  The  third  optimization  maxi¬ 
mizes  the  revenue  of  the  toll  collecting  agency,  n^  ■  nQ  ■  ng  is  the 
maximum  number  allowed  in  the  system.  Thus,  if  left  on  its  own,  the 
toll  collecting  agency  levies  tolls  that  are  too  high  for  the  social 
good. 

Prabhu  (1974)  considers  the  problem  of  finding  an  optimal 
stopping  time  for  an  M/G/l  queueing  system  with  a  constant  arrival 
rate.  The  stopping  time  Is  chosen  to  maximize  expected  discounted 
profit  when  the  cost  structure  is  as  follows: 

a)  A  revenue  per  unit  time. 

b)  An  operating  cost  per  unit  time  in  (0,  t  ) ,  where  t  is 

o  o 

the  stopping  time. 

c)  An  operating  cost  rate  in  (t  ,  °°) . 

o 

Finding  the  optimal  rtopping  time  is  based  on  an  infinitesimal  look¬ 
ahead  rule  which  can  be  described  as  follows.  Suppose  the  system  has 
been  operating  to  time  t  and  a  profit  f(t,  Wt>  has  been  made,  where 
is  the  remaining  workload.  It  is  profitable  to  continue  the  operation 
up  to  time  t  +  h  if  E{f(t  +  h,  W  +  ^))  >  f(t,  Wt> . 
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Stidham  (1978)  considers  the  problem  of  accepting  or  rejecting 
arrivals  at  a  GI/M/1  queueing  system.  The  holding  cost  is  convex  in 
the  number  of  customers  in  the  system  and  the  reward  for  service  is  a 
random  variable.  Finite  and  infinite  horizon  problems  with  and  without 
discounting  are  considered.  A  terminal  reward  or  cost  is  allowed  in 
the  finite  horizon  problems.  The  author  shows  that  a  socially  optimal 
policy  is  less  likely  to  accept  a  customer  than  an  individually 
optimal  policy. 

Stidham  and  Prabhu  (1974)  examine  the  work  that  has  been  done  in 
control  of  queueing  systems.  The  authors  show  how  several  of  the  works 
are  related  and  point  out  some  generalizations  that  apply  to  most  of 
the  papers.  For  instance,  they  note  that  most  research  has  sought  to 
determine  1)  when  a  stationary  policy  is  optimal,  2)  its  form,  and 
3)  the  value  of  its  parameters. 

Yechiall  (1971)  considers  a  GI/M/1  queueing  system.  The  cost 
structure  includes  a  reward  for  service,  a  service  charge  for  each 
customer  served,  a  cost  per  unit  time  in  the  system,  and  a  charge  for 
balking.  The  author  finds  an  individual  optimum  that  is  analogous  to 
that  of  Naor  (1969) .  He  treats  the  social  optimum  problem  as  a  semi- 
Markov  decision  process  and  shows  that  a  control-limit  rule  is 
optimal;  that  is,  the  administrator  will  operate  the  queue  as  a  finite 
capacity  queue.  As  Naor  did,  he  shows  that  the  social  optimum  is  less 
than  or  equal  to  the  individual  optimum  balking  point.  He  then  formu¬ 
lates  a  linear  program,  equivalent  to  the  social  optimum  problem.  The 
author  also  considers  the  revenue  maximizer's  optimum. 


Yechiali  (1972)  extends  the  results  of  Yechiali  (1971)  to  a 
GI/M/s  queueing  system.  The  author  shows  that  the  social  optimum 
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balking  point  is  less  than  or  equal  to  the  individual  optimum  using 
the  same  techniques  as  in  his  earlier  paper. 

2.2  Control  of  Server 

Balachandran  (1971)  considers  an  M/G/l  queueing  system  with  two 
classes  of  arrivals.  Class  one  customers  are  of  higher  priority  than 
class  two  customers.  When  N  class  one  customers  are  in  the  queue,  the 
service  of  a  class  two  customer  is  preempted  until  all  N  class  one 
customers  are  served.  Service  of  the  class  two  customer  is  then 
resumed  without  loss  of  service  accomplished.  The  value  of  N  is  sought 
which  minimizes  a  linear  cost  function  involving  the  expected  number  of 
customers  of  each  type  and  the  expected  preemption  rate. 

Crabill  (1972)  examines  an  M/M/1  queueing  system  with  a  constant 
arrival  rate  and  a  service  rate  y^  where  u^ety^,  ...»  y^,}.  A  policy  for 
choosing  the  service  rate  as  a  function  of  queue  length  is  sought  to 
minimize  the  long-run  average  expected  cost  rate  of  the  queue.  The 
cost  expression  involves  a  customer  inconvenience  cost  rate  which 
depends  on  the  number  in  the  system  and  a  service  cost  rate  which 
depends  on  the  service  rate  used.  The  optimal  policy  is  shown  to  be  a 
set  of  k  +  .]  numbers  which  specify  the  range  of  values  of  number  in  the 
system  for  which  each  service  rate  is  to  be  used. 

Heyman  (1968)  considers  an  M/G/l  queueing  system  with  the 
following  cost  structure: 

a)  Dormant  cost  rate. 

b)  Running  cost  rate. 

c)  Start  up  cost. 

d)  Shut  down  cost. 
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e)  Holding  cost  rate. 

The  purpose  of  the  paper  is  to  find  the  policy  that  minimizes  the  total 
expected  operating  cost  from  among  all  possible  policies  of  turning  the 
server  off  or  on  during  the  operating  horizon.  For  the  undiscounted 
infinite  horizon  model,  the  optimal  policy  is  shown  to  be  either  the 
server  is  always  running  or  the  server  is  turned  on  when  n  customers 
are  in  the  system  and  off  when  the  system  is  empty.  For  small  interest 
rates,  the  optimal  discounted  policy  is  approximately  the  same  as  the 
optimal  undiscounted  policy.  The  author  gives  a  recursive  relation  for 
determining  the  optimal  policy  for  a  finite  horizon. 

Zacks  and  Yadin  (1969)  examine  an  M/M/1  queueing  system  with 
arrival  rate  X  and  service  rate  y,  where  y  can  be  chosen  from  (0,y*). 

The  following  cost  structure  is  considered: 

a)  Holding  cost  rate. 

b)  Cost  of  switching  service  rate. 

c)  Service  cost  rate. 

The  policy  specifying  service  rates  which  minimizes  discounted  costs 
over  an  infinite  horizon  is  sought.  The  optimal  control  times  are 
shown  to  occur  immediately  after  a  change  in  queue  size.  For  the 
special  case  of  linear  holding  costs,  switching  costs  that  depend  only 
on  the  absolute  value  of  the  change  in  service  rate,  and  convex  service 
cost  rates  with  bounded  derivatives,  the  optimal  control  policy  is  of 
the  form: 

a)  Increase  y  to  r^(x)  if  y  <  r^(x) . 

b)  No  change  if  j}(x)  ■  y  ■  n(x). 

c)  Decrease  y  to  n(x)  if  y  >  n(x). 

l 

I 
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x  is  the  number  in  the  system  and  Jl(x)  and  ri(x)  are  service  rates 
associated  with  a  given  number,  x,  in  the  system. 

Zacks  and  Yadin  (1970)  examine  the  special  case  of  their  earlier 
paper  in  a  little  more  detail.  Some  additional  properties  of  the 
optimal  policy  are  established  and  a  numerical  example  is  provided. 

2.3  General 

Dear  (1976)  considers  the  problem  of  scheduling  the  landing  of 
aircraft.  He  constrains  the  capability  of  the  scheduler  so  that  he  can 
move  an  aircraft  no  more  than  a  given  number  of  positions  from  its 
first-come  first-served  position.  These  constraints  keep  the  schedule 
feasible  in  that  changes  causect  by  a  new  arrival  will  be  relatively 
minor  and  any  given  aircraft  will  land  reasonably  close  to  its  first- 
come  first-served  time.  Yet  the  constraints  are  loose  enough  to  allow 
considerable  improvement  in  runway  utilization  over  the  first-come 
first-served  schedule. 

Hddgson  and  Koehler  (1978)  examine  the  solution  of  Markov 
decision  processes  by  policy  iteration  when  the  systems  require  large 
numbers  of  states.  The  authors  give  a  procedure  for  transforming 
finite  state,  continuous  time  Markov  or  semi-Markov  decision  processes 
so  that  an  approximation  algorithm  by  White  and  Odoni  can  be  used. 

Koopman  (1972)  considers  queues  of  aircraft  awaiting  landing  at 

a  single  runway  where  there  is  a  fixed  maximum  queue  length.  Arrivals 

are  assumed  to  form  a  time  dependent  Poisson  process  and  service  times 

are  either  time  dependent  exponential  random  variables  or  time 

dependent  constants.  A  periodic  solution  is  assumed  for  the  probability 

of  n  aircraft  in  the  system;  that  is,  P  (t  +  T)  -  P  (t) .  Numerical  1 

n  n 

I  1 
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results  are  given  for  several  sets  of  values  of  the  parameters.  For 
the  cases  run,  the  author  notes  that  results  such  as  expected  number 
in  the  system  were  remarkably  insensitive  to  the  form  of  the  distribu¬ 
tion  assumed  for  service  times. 

White,  Schmidt,  and  Bennett  (1975)  present  in  Chapter  VII  the 
statistical  techniques  necessary  for  modeling  the  components  of 
queueing  systems.  Modeling  the  arrival  and  service  distributions  are 
of  major  interest.  The  authors  give  the  theory  and  examples  for: 

a)  Selecting  candidate  distributions. 

b)  Determining  numerical  values  for  the  parameters  of  the 
distributions . 

c)  Testing  the  hypothesis  that  the  chosen  distribution  is 
the  true  distribution. 


CHAPTER  III 


THE  SINGLE  CLASS  MODEL 

This  chapter  defines  Model  I,  the  single  class  of  customers 
model.  The  methods  that  Naor  (1969)  and  Yechiali  (1971)  used  to  solve 
for  the  social  optimum  are  presented  and  their  approaches  are  compared. 
The  model  is  also  formulated  as  a  linear  program  following  Yechiali 
(1971) .  Then,  the  effect  of  changing  the  arrival  rate  on  the  solution 
is  investigated.  Finally,  entry  tolls  are  used  to  impose  the  social 
optimum  solution  on  individual  customers. 

3.1  Model  I 

Model  I  has  a  single  class  of  customers.  The  arrivals  form  a 
Poisson  stream  with  mean  rate  X.  The  service  times  of  the  single 
server  are  independent,  identically  distributed,  exponential  randou 
variables  with  mean  1/y.  Thus,  this  model  is  an  M/M/1  queueing  model. 
The  following  cost  structure  is  Imposed  on  the  operation  of  the 
queueing  system: 

a)  Each  customer  served  receives  a  reward  of  R  dollars. 

b)  Each  unit  of  time  a  customer  spends  in  the  system  costs 
him  C  dollars. 

Suppose  that  etch  arriving  customer  is  given  the  choice  of 
joining  the  queue  and  receiving  reward  R  and  paying  C  per  unit  time 
in  the  system  or  of  not  joining  and  not  paying  or  receiving  any  money. 
In  this  and  the  other  models  to  be  considered,  customers  are  assumed  to 
decide  by  comparing  the  expected  net  gain  associated  with  each  decision 
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and  choosing  the  action  with  the  larger  gain.  (In  case  of  a  tie,  the 
customer  joins  the  queue.)  This  model  is  considered  by  Naor  (1969) 
and  Yechiali  (1971) ,  although  Yechiali  allows  a  general  arrival 
distribution  and  a  slightly  more  general  cost  structure. 

3.2  Naor's  Approach 

After  detailing  the  assumptions  and  structure  of  the  model,  Naor 

argues  that  all  reasonable  strategies  lead  to  a  finite  capacity  queue. 

The  task  remaining  then  is  to  determine  the  optimal  capacity,  n.  Naor 

first  deals  with  determining  n  ■  n  under  self-optimization  where  each 

s 

customer  considers  only  his  own  expected  net  gain  in  deciding  whether 

or  not  to  join  the  queue.  The  expected  net  gain  for  joining  is 

R  -  (i  +  l)C/y,  where  i  is  the  number  of  customers  the  arrival  finds 

in  the  system.  Joining  the  queue  serves  the  self-interest  of  a 

customer  if  i  is  lees  than  n  ,  where  n  is  such  that 

s  s 

R  -  (n  +l)C/y  <  0  -  R  -  n  C/u  . 
s  s 

This  strategy  leads  to  an  M/M/l/n  queueing  system  where 

S 

ng  •  [Ru/c]  .  (3.1) 

The  brackets  indicate  that  n  is  the  greatest  integer  in  Ru/C.  Thus, 

s 

the  capacity  of  the  queue  or  the  balking  point  is  n  for  the  self- 

s 

optimization  problem. 

If  each  customer  or  an  administrator  acts  to  maximize  the  sum 
of  the  individual  net  benefits,  the  problem  becomes  the  social  optimum 
problem.  Considering  an  infinite  horizon  without  discounting,  Naor 
sets  up  an  expected  overall  net  benefit  rate  function.  In  notation 
based  on  Gross  and  Harris  (1974),  the  expected  net  benefit  rate  is 


a--1' 


as  follows: 
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g(n)  -  X' (n)R  -  CL(n)  ,  (3.2) 

where  X’(n)  and  L(n)  are,  respectively,  the  effective  arrival  rate  and 
the  expected  number  in  the  system  when  a  maximum  of  n  customers  is 
allowed  in  the  system.  g(n)  is  the  expected  net  benefit  rate  or 
expected  rate  of  gain  when  a  maximum  of  n  customers  is  allowed  in  the 

system.  The  units  of  g(n)  are  dollars  per  unit  time.  Setting  p  -  X/y 

and  using  formulas  for  X'(n)  and  L (n)  that  are  available  in  Gross  and 
Harris  (197A)  yields: 

g(n)  -  Xr{1  -  pn(l  -  p)/(l  -  pn+1)}  -  C<p/(1  -p)~ 

-  (n  +  l)pn+1/(l  -  pn+1)}  .  (3.3) 

The  administrator  wants  to  choose  n  to  maximize  g(n).  g(n)  is 

discretely  unlmodal  in  n  (proof  given  in  Appendix  B) ,  which  implies 

that  a  local  maximum  is  a  global  maximum.  Thus,  the  administrator 

needs  to  find  n  ■  n  ,  such  that: 

o 

Ag(nQ  +  1)  <  0  -  Ag(nQ)  , 

where  Ag(n)  ■  g(n)  -  g(n  -  1)  .  Naor  shows  that  nQ  satisfies 

{nQ(l  -  p)  -  p(l  -  pn°)}/(l  -  p)2 

-  Ry/C  <  {(nQ  +  1)(1  -  p)  -  p(l  -  Pn°+1)}/(1  -  p)2  . 

(3. A) 

For  continuous  variables  v  ■  Ry/C,  and  v  related  by 

s  o  J 

{vo(l  -  p)  -  p(i  -  pV°)}/(l  -  p)2  -  v8  ,  (3.5) 

Equation  (3. A)  leads  to  v  «  v  and,  since  v  increases  with  v  , 
n  o  s  s  o 

nQ  -  [vqJ«  n^  »  [v  ]  .  (Proofs  of  these  assertions  also  appear  in 
Appendix  B.) 
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Lippman  and  Stidham  (1977)  and  others  assert  that  nQ  ■  ng  Is 
due  to  an  individual  optimizer’s  failure  to  consider  the  decrease  in 
benefits  to  later  arriving  customers  caused  by  his  joining  the  queue. 
This  type  of  effect  is  called  an  external  economic  effect.  Considered 
another  way,  the  difference  between  the  social  optimum  formulation  and 
the  individual  optimum  formulation  is  that  the  social  optimum  formula- 
tion  Includes  arrival  rata  information.  This  information  allows  the 
administrator  to  anticipate  expected  benefits  from  customers  who  have 
yet  to  arrive. 

The  queue  capacity  of  the  Individual  optimum  problem  serves  as 
an  upper  bound  on  that  of  the  social  optimum  problem.  This  bound  is 
useful  in  limiting  the  number  of  possible  solutions  that  need  to  be 
considered  when  approaching  the  problem  as  Yechlall  does.  The  optimal 
capacity  nQ  of  the  social  optimum  problem  is  found  by  solving  Equation 
(3.5)  for  vqI  then  setting  nQ  ■  [vq] .  Thus,  Naor  presents  solutions  to 
both  the  individual  and  social  optimum  problems. 

3.3  Yechlall1 a  Approach 

Although  Yechlall  (1971)  considers  a  general  Interarrival  time 

distribution  and  a  slightly  more  general  cost  structure  than  Model  1, 

this  discussion  of  his  results  is  couched  in  the  terminology  and  form 

of  Model  1.  First,  a  few  terms  and  concepts  need  to  be  defined.  nn  Is 

the  number  of  customers  in  the  system  at  the  Instant  of  the  n  arrival. 

{A^}  is  the  sequence  of  successive  decisions  made  by  arriving  customers, 

where  A  ■  0  if  the  n^  customer  balks,  and  A  ■  1  if  he  joins  the 
n  n 

queue,  (n  ,  A  } ,  n  •  1,  2,  . . .  is  a  semi-Markov  decision  process, 
n  n 

(See  Appendix  C  for  a  brief  introduction  to  semi-Markov  decision 
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processes.)  Hn  «  (r^,  ^ . A^'H^  -  0)  is  the  history  of  the 

process  through  arrival  m.  A  policy  P  for  controlling  the  system  is  a 

set  of  decisions  ^(H^,  t^)},  m  -  1,  2,  . . . ,  k  -  0,  1.  D^H^,  y 

is  interpreted  as  the  probability  of  implementing  decision  k  (k  ■  0  if 

balk,  1  if  Join)  at  time  m  given  history  H  .  and  present  stawe  n  when 

m-i  m 

policy  P  is  in  effect. 

Yechiali  categorizes  possible  policies  for  controlling  the 

system  in  the  following  manner.  Let  c  be  the  class  of  all  policies 

^Dk^Hm-l'  Let  Cs»  a  subclass  of  Ct»  be  the  class  of  stationary 

Markovian  policies;  that  is,  for  each  arrival,  only  the  state  of  the 

system  at  the  instant  of  the  arrival  is  used  as  a  basis  for  making  the 

decision  of  whether  or  not  to  Join.  Since,  in  Model  I,  the  service 

times  are  exponential  and  the  horizon  is  infinite,  only  stationary 

Markovian  policies  need  to  be  considered.  For  convenience,  let  D^(l) 

be  the  stationary  probability  that  A  -  1  given  that  n  -  i.  and 

n  'n 

{l  -  D^(l) }  -  D^(0)  be  the  stationary  probability  that  Aq  -  0  given 

that  rin  “  1.  is  the  class  of  all  stationary  policies  such  that 

0  <  D1(l)  -  1  for  i  -  k  and  D^l)  -  0  for  i  >  k.  Sk  is  called  the 

class  of  stationary  control-limit  policies  of  order  k.  Let  S  denote 

the  class  of  stationary  control-limit  policies  of  infinite  order;  thus, 

PeS  if  P  ■  (D^(l) :  D^(l)  >  0,  i  ■  0,  1,  ...}.  Yechiali  calls  the  union 

of  S.  and  S  the  class  of  control-limit  policies,  denoted  by  C  ,  which 

ca 

is  a  subclass  of  C^.  Let  C^,  a  subclass  of  C^,  be  the  class  of 
nonrandomized  policies,  thus,  PeCd  implies  d£(.,0  -  0  or  1.  The 
class  of  deterministic  control-limit  rules,  C^,  is  a  subclass  of  Cd 
and  Ccr  Pr.CdcJl  if  P  -  {D1(l):  D^l)  -  1,  i  -  k;  DjU)  -  0,  i  >  k} 
for  some  k  or  P  -  {D±(l) :  D±(l)  -  1,  i  -  0,  1,  ...}. 


Now,  consider  the  individual  optimum  problem.  A  policy,  P,  or 

p 

a  set  of  joining  probabilities  is  sought  that  maximizes  an 

arrival's  expected  net  benefit.  If  the  customer  balks,  his  expected 
net  benefit  is  zero.  Since  the  service  times  are  exponential,  the 
expected  time  in  the  system  for  an  arrival  who  finds  i  in  the  system 
is  (i  +  l)/y.  Thus,  the  expected  net  benefit  for  joining  is 
R  ~  (i  +  l)C/y.  A  policy  P  is  sought  such  that 

oo 

l  D*(H  ,,i)(R  -  (i  +  l)C/y)  (3.6) 

i-o  1  m_1 

is  maximized.  If  n  is  such  that  R  -  n  C/y  ■  0,  but  R  -  (n  +  l)C/y  <0, 

s  s  s 

then  clearly  P  -  {D?(H  ,,i)  *  1,  i  <  n  ;  dJ(H  . ,i)  -  0,  i  -  n  }  is 

x  m— i  s  x  tu-"  u.  s 

the  policy  that  maximizes  Equation  (3.6).  The  optimal  policy  is  a 
deterministic  control-limit  rule  which  is  indeed  the  same  policy  that 
Naor  found. 

ft 

Determining  the  optimal  policy  for  the  social  optimum  problem  is 
more  involved.  The  customers  or  administrators  are  assumed  to  act  to 
maximize  the  expected  net  benefit  per  unit  time  of  the  arrivals  as  a 
whole.  If  g  is  the  expected  net  benefit  rate,  two  policies  P  and  P' 
are  called  g-equivalent  if  g^  »  gpt.  The  expected  net  gain  rate  of  the 
arrivals  as  a  whole  under  policy  P  is: 

OO 

gp  -  l  dJ(1)X(R  -  C(i  +  l)/y}  ,  (3.7) 

i-o 

p 

where  0^  is  the  steady  state  or  stationary  probability  that  the  system 

p 

is  in  state  i  if  policy  P  is  used.  D^(l)  is  the  stationary  probability 
that  customers  are  allowed  to  join  under  policy  P  when  i  customers  are 
in  the  system.  The  main  difference  between  the  social  optimum  formula¬ 
tion  and  the  individual  optimum  formulation  is  that  the  social  optimum 


formulation  includes  informatioi  about  the  stationary  probabilities  of 
the  various  states.  This  information  allows  the  administrator  to 
anticipate  expected  benefits  from  customers  who  have  yet  to  arrive. 

Yechiali  first  establishes  that  for  every  PeCg,  there  is  a 

g-equivalent  rule  P'eC^.  Thus,  the  search  for  the  solution  to  the 

social  optimum  problem  can  be  restricted  to  control-limit  policies. 

He  uses  the  fact  that  a  nonrandomized  policy  is  optimal  for  a  finite 

state  space  to  show  that  a  deterministic  control-limit  policy,  PeC^^, 

is  optimal  for  the  finite  state  spac-d  social  optimum  problem.  For  the 

infinite  state  space  social  optimum  problem,  he  considers  two  cases, 

an  ergodic  case  and  a  nonargodic  case.  For  the  nonergodic  case  (X  *  y) , 

he  argues  that  L(n)  -*■  °°  as  n  -*■  thus,  g(n)  •+•  -  00  as  n  -*•  ®.  (Here, 

g(n)  denotes  the  expected  gain  rate  under  any  control-limit  policy 

with  a  maximum  of  n  customers  in  the  system.)  Since  g(n)  is  finite 

when  n  is  finite,  the  finite  state-space  result  can  be  used  to  show 

that  a  finite  deterministic  control-limit  policy  is  optimal  for  the 

nonergodic  case.  In  the  ergodic  case  (X  <  y) ,  Yechiali  uses  the  limit 

of  a  sequence  of  finite  state-space  problems  to  show  that  again  a 

deterministic  control-limit  policy  Is  optimal.  He  then  uses  arguments 

based  on  the  policy  iteration  algorithm  of  Howard  (1960)  to  establish 

that  the  social  control-limit,  n  -  1,  is  less  than  or  equal  to  the 

o 

individual  control-limit,  n  -  1.  (Howard's  policy  iteration  algorithm 

s 

is  discussed  in  Appendix  C.) 

The  solution  to  the  social  optimum  problem  can  be  found  using 
Howard's  policy  iteration  algorithm.  (The  solution  of  a  small  problem 
by  policy  iteration  is  given  in  Appendix  C.)  For  each  state  i,  only  two 
choices  exist  in  the  policy  iteration  procedure,  either  reject  the 
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customer,  k  ■  0,  or  accept  the  customer,  k  -  1.  For  later  models,  the 

range  of  alternatives  will  expand.  When  policy  iteration  is  used  to 

solve  for  n  in  Yechiali's  formulation,  n  serves  as  an  upper  bound  on 
o  s 

the  state  space  needed  to  describe  the  problem. 

3.4  Comparison  of  Results 

The  two  developments  of  n  ,  the  balking  point  in  the  self- 

3 

optimum  problem,  given  in  Sections  3.2  and  3.3,  lead  to  the  same 
expression  for  determining  n  ;  namely,  n  -  [Ry/cJ .  This  section 
establishes  that  the  gain  rate  expressions  for  the  social  optimum 
problem  (and  thus,  the  solutions)  in  Sections  3.2  and  3.3  are  the  same. 
In  fact,  the  expressions  are  also  shown  to  be  equivalent  to  a  third 
gain  rate  expression. 

Naor's  work  determines  the  capacity  of  the  system,  nQ,  that 
maximizes  Equation  (3.2).  Yechiali’s  work  determines  a  control-limit 
policy  that  maximizes  Equation  (3.7).  Control  limits  and  system 
capacities  are  related  in  the  following  manner:  a  control  limit  of  j 
leads  to  a  capacity  of  j  +  1.  A  third  term  used  here,  balking  point 
or  forced  balking  point,  is  equivalent  to  the  capacity  of  the  system. 

Let  Naor's  form  of  the  expected  social  gain  rate  be  called  g^. 
From  Equation  (3.2),  g^(n)  *  X'(n)R  -  L(n)C.  g^(n)  is  Yechiali's  form 
of  the  expected  gain  rate  expression.  In  this  form  of  the  gain  rate 
expression,  the  expected  net  reward,  i.e.,  the  reward  for  service  minus 
the  expected  waiting  cost  (R  -  C(i  +  l)/y}  is  given  to  an  arrival  the 
instant  he  joins  the  system.  In  the  third  form,  g^(n) ,  the  waiting 
costs  are  charged  over  the  length  of  the  customer's  stay  in  the  system 
while  his  reward  is  given  to  him  when  he  joins.  g^(n)  and  g^(n)  are 
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developed  using  notation  from  Howard  (1971)  and  both  are  shown  to  be 
equal  to  &N(n) . 

g,j,(n)  is  developed  first,  n,  the  forced  balking  point,  is  the 
upper  limit  on  the  number  in  the  system.  Lee  y^  be  the  cost  per  unit 
time  the  system  is  in  state  i,  where  state  i  indicates  that  i  customers 
are  in  the  system.  Here,  y^  is  simply  Ci.  Let  b.,.,  be  the  reward  for  a 
transition  from  state  i  to  state  j. 

R  j  ™  i  +  1 

**ij  *  (0  otherwise 

l 

Let  be  the  expected  holding  time  in  state  i  given  that  the  next 

transition  ia  to  state  j.  P . .  is  the  probability  of  a  transition  from 
—  n  „ 

state  i  to  J .  T.  «  1  P.  .t,  .  is  the  expected  waiting  time  in  state  i. 

1  j«0 

If  is  defined  to  be  the  expected  reward  per  occupancy  of  state  i, 
then  s 


n 


h  -  J.o  Vv« +  V 


-clTi  +  pi,i+ia 


Define  to  he  the  expected  reward  per  unit  time  in  state  i. 


qi  ■  ri/Ti  *  "0i  +  pi,i+iR/Ti 


S  neft  n  is  the  forced  balking  point,  a  state  dependent  arrival  rate  X^ 

may  be  defined  as  follows: 

1 

X  i  <  r. 

X  m 

i  > 

x  0  i  -  n  . 

> 

Since  the  interarrival  and  service  time  distributions  are  exponential, 
the  transition  rate  out  of  state  i,  T^,  is  given  by: 
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' 

X  i  -  0 

T  -  1 

1  \±  +  u  i  >  0 

» 

Thus,  T  -  1/T^  or 

1/Xi  i  -  0 

Ti  "  ‘  l/(Xi  +  u)  i  >  0 

Also,  the  probability  of  a  transition  from  state  i  to  i  +  1  is 

1  i  -  0 

Pi,l+1  ■  '  Y<Xi  +  »>  0  <  i  <  n 

0  i  -  n 

» 

Thus,  the  expected  reward  rate  in  state  i  becomes 
qi  «  -  Ci  +  XaR 

Using  results  from  Howard  (1960),  gT(n)  can  be  written  as 
n 

gT(n)  -  l  0i(n)  q^,  ,  (3.8) 

i-o 

where  0^(n)  is  the  steady  state  probability  that  i  customers  are  in  the 
system  given  that  the  forced  balking  point  is  n.  (This  equation  is 
similar  to  Equation  (3.7)  in  Section  3.3.)  Thus, 

gT(n)  *  l  0, (n)X.R  -  C  l  0 . (n)  i 
i-o  1  i-o 

n-1 

-  XR  l  0  (n)  -  CL(n) 
i-o 

n-1 

X  J  0 , (n)  is  the  mean  rate  at  which  customers  actually  join  the 
i-o  1 

system  which  is  the  effective  arrival  rate,  X'(n).  Thus, 
gT(n)  -  X'(n)R-  CL(n)  -  gN(n) 

> 

i  3 

n 

x- 
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For  &y(n) ,  y  -  0  and 

R  -  (i  +  l)C/y  j  -  i  +  1 

b  m 

ij  0  otherwise 


Thus, 


r 


1 


|.o  V«  •  -  «  +  • 


The  expected  reward  per  unit  time  In  state  1  Is  given  by 


q±  -  -  Pi,i+1^R  "  +  DC/V}/^ 

X^  and  T  for  g^(n)  are  the  same  as  they  were  for  gT(n) .  Thus, 

1{R  -  (1  +  DC/yJ/U/X^  1-0 

(Xi/(Xi  +  y)}{R-  (i  +  l)C/y}/{l/(Xi  +  y}  1  >  0 


Thus,  for  any  1,  q^  -  X^R  -  X^(i  4-  l)C/y.  As  in  Equation  (3.8), 
n 

g«(n)  -  l  0  (n)q., 

1  i-o  1 

which  becomes 


n-1  n-1 

g^n)  -  XR l  0i(n)  -  c£  (X/y)0i(n)(i  +  1) 

i-o  i-o 

For  an  M/M/l/n  queue,  0^+1(n)  *  (X/y)0i(n)  if  1-  1  +  1  -  n.  Thus, 

n-1 

g^n)  -  X'  (n)R  -  cj  01+1(n)(i  +  l) 

i-o 

-  X'(n)R  -  CL(n) 

'  Mn) 

and  this  establishes 


Theorem  3.1:  gN(n)  ■  g^a)  “  8T(“)  • 

(A  result  analogous  to  gy(n)  -  g^(n)  for  a  discounted  Infinite  horizon 
problem  is  established  in  Stidham  and  Prabhu  (1974).) 
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3.5  Linear  Programming  Formulation  of  the  Social  Optimum  Problem 

Naor’s  formulation  of  the  social  optimum  problem  can  be  written 
as 

m^X  g(n)  “  X'(n)R  -  CL(n) 

which  is  a  nonlinear  programming  problem  with  an  Integer  valued 

decision  variable.  However,  this  section  shows  that  Yechiali'3 

formulation  can  be  transformed  into  a  linear  programming  problem. 

The  following  linear  programming  formulation  is  similar  to  that 

of  Yechiali  (1971)  but  is  modified  to  maximize  expected  gain  rate  as  in 

Fox  (1966).  Since  n^  ■  n^  only  states  0  through  n^  can  have  positive 

steady  state  probabilities.  A  policy  P*eC  is  sought  such  that 

s 

ns  1  ns  1 

Sp*  -  {I  l  0*D*(k)r  (k>}/{£  l  0?D^(k)T.(k)}.(3.9) 
a  i-o  k-o  1  1  1  i-ok-o  1 

p 

0^  is  the  steady  state  probability  that  i  are  in  the  system  under 

P 

policy  P.  Di(k)  is  the  probability  of  choosing  action  k  (k  -  0  if 
balk,  1  if  join)  in  state  i  when  policy  P  is  used.  r^(k)  is  the 
expected  reward  per  occupancy  of  state  i  if  action  k  is  chosen,  ^(k) 
is  the  expected  waiting  time  in  state  i  when  action  k  is  chosen. 

Equation  (3.9)  is  just  Equation  (3.7)  written  in  a  slightly 
different  manner.  D^(1)A{r  -  (i  +  l)C/y)  from  Equation  (3.7)  is  q.  (1), 
the  expected  gain  per  unit  time  in  state  i  when  customers  are  allowed 
to  join.  However,  q^l)  -  r^D/t^l)  and  qi(0)  -  0.  In  Equation 
(3.9),  the  r^(k)  and  q^(k)  terms  are  summed  separately,  then  divided, 
as  opposed  to  Equation  (3.7)  where  q^(k)  is  formed  first.  In  words, 
Equation  (3.9)  is  the  expected  gain  per  transition  divided  by  the 
expected  time  per  transition  which  is  indeed  the  expected  gain  per 
unit  time. 


The  balance  equations  and  normalizing  equation  which  the  steady 
P  P 

state  probabilities  10^D^(k)}  must  satisfy  are 


n  -  1 
s 


(3.10) 


and 

n  1 

f  l  0jD^(k)  -  1  (3.11) 

j*o  k“o  2  2 

where  0  *  0^  -  1  and  0  *  D^(k)  ■  1.  P^(k)  is  c^e  probability  of  a 

transition  from  state  i  to  j  if  action  k  is  chosen.  Note  that  Equation 

(3.10)  consists  of  only  n  equations  since  the  redundant  balance 

s 

equation  for  state  n  has  been  dropped  from  the  formulation.  Equations 

s 

(3.9)  to  (3.11)  constitute  a  linear  program  with  a  fractional  objective 
function. 

A  linear  program  equivalent  to  Equations  (3.9)  to  (3.11)  but 
without  the  fractional  objective  function  will  be  developed  using  the 
work  of  Fox  (1966)  and  Charnes  and  Cooper  (1962).  First,  let 
x^(k)  ■  0^D^(k),  so  that  Equations  (3.9)  to  (3.11)  become 

ns  1  ns  1 

l  x^(k)ri(k)}/{J  l  x^(k)x ,  (k) } 
s  i*o  k*o  i«o  k”0 

subject  to 


. . . , 


n  -  1 
s 


s  1 

I  l  x*(k)  -  1  x*(k)  -  0 

j-o  k-o  2  2 
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Let  y^(k)  *  tx^k),  where  t  ■  0,  be  chosen  so  that 


I  l  y!(k)  t  (k)  -  l 
i-o  k-o 

Multiplication  of  the  objective  function  by  tft  yields 

£  'f  \  • 

s  i-o  k-o 

After  multiplication  of  Equations  (3.10)  and  (3.11)  by  t,  the  linear 
program  becomes 


™  l  I 

s  i-o  k-o 

subject  to 


l  l  yj(k)  P..(k)  -  l  y?  (k)  -  0,  j  -  0,  ng  -  1 

i-o  k-o  J  k-o  J 


IS  l  yj(k)  -  t 

j-o  k-o  J 


l  l  yJ(k)T.(k)  -  l 


j-o  k-o 


'j'  '  j 


y^(k)  -  0,  t  -  0 


The  last  constraint  is  included  to  maintain  the  transformation.  The 

first  n  constraints  together  with  the  last  allow  at  most  n  +  1  of 
s  s 

the  yj(k)'s  to  be  positive.  The  next  to  last  constraint  merely  adjusts 
the  value  of  t.  Since  t  is  of  no  interest,  it  and  the  next  to  last 
constraint  can  be  dropped  from  the  formulation.  Thus,  Equations  (3.9) 
to  (3.11)  can  be  written  as  the  following  equivalent  linear  program: 
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s  1 

max  I  l  y^lOr^k) 


(3.12) 


i-o  k-o 


s  1 


subject  to 
1 


I  I  y,  OOP,,  00  -  l  y,00  -  o,  j 

i*o  k*o  1  k*o  J 


44  1 

l  l  y,  00?,  (k)  -  l 

j«o  k-o  2  2 

yd00  ^  o 


0,  •  •  «  )  Q  “  1 

s 

(3.13) 

(3.14) 

(3.15) 


The  superscript  P  has  been  dropped  since  the  optimal  policy  can  be 
recovered  from  the  solution  to  Equations  (3.12)  to  (3.15)  through 


Di*  (k)  “  yi(k)/ 


L 


y, (k) »  i  —  0,  ...»  n  i  k  —  0,  1. 


(3.16) 


JP* 


Fox  also  shows  that  the  Di  (k)  are  either  zero  or  one  and  at  most 

pj|f 

n  +  1  of  them  are  one.  0.  can  be  found  from 
s  i 


,P* 


yfOO  T±(k) , 
0  . 


yt(k)  >0 
otherwise 


The  linear  program  yields  the  social  balking  point  through 

nQ  -  max{i:  y^(l)  >  0}  +  1  ,  (3.17) 

3.6  g  and  as  Functions  of  \ 

It  seems  reasonable  to  conjecture  that  the  expected  gain  rate 
can  only  increase  as  the  arrival  rate  increases.  This  is  now 
established. 
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Theorem  3.2:  g  is  a  nondecreasing  function  of  X. 

Proof:  Let  X1  >  X".  Define  g(P,X)  to  be  the  expected 

gain  rate  under  policy  P  when  the  arrival  rate  is  X. 

Let  g*(X)  denote  the  optimal  expected  gain  rate  when  the 

arrival  rate  is  X.  Since  Yechiali  (1971)  shows  that  a 

deterministic  control-limit  policy  is  optimal,  let 

P*  •  (D . (1) :  D.(l)  -  1,  i  -  n  -  1;  D,(l)  «  0,  1  >  n  -  1} 

11  o  1  o 

be  the  optimal  policy  when  the  arrival  rate  is  X".  Let 

P'eCcJl  (the  set  of  stationary  control-limit  policies)  be 

such  that  P'  -  (D. (1):  D.(l)  ■  X"/  X  ' ,  i  ■  n  -  1; 

ll  o 

D^(l)  *0,  i  >  n  -  l).  Under  policy  P’,  an  arrival  who 
finds  less  than  nQ  in  the  system  is  allowed  to  join  with 
probability  X"/  X 1  and  is  forced  to  balk  with  probability 
1  -  (X"/X').  Thus,  P'  is  not  a  deterministic  policy. 

Since  no  penalty  is  assessed  for  rejecting  a  customer, 
g(P',  X’)  -  g(P*,  X").  Finally,  since  g*(X f )  -  g(P',X’), 
g*(X’)  -  g*(X"). 

Although  policy  iteration  gives  a  method  for  determining  the 
social  optimum,  nQ,  for  a  given  X,  the  following  section  presents  a 
method  for  determining  the  range  of  values  of  X  over  which  a  given  nQ 
is  optional. 

Define  {f(i)}  to  be  the  sequence  of  expected  net  rewards  of 
joining  customers,  where  f(i)  *  R  -  C(i  +  l)/y  is  the  expected  net 
reward  if  1  customers  are  in  the  system.  In  Yechiali 's  formulation, 
an  arriving  customer  receives  his  expected  net  reward  upon  entering 
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Che  system.  Thus,  for  Yechi.xli's  formulation,  the  expected  net  gain 
of  the  system  per  occupancy  of  state  1  Is 


ri- 


Since 


where 


'f(i) 

o 

■ 

.{Ai/(Ai  +  y) }f (i) , 

i  >  0 

fl/Xi  . 

i  -  0 

11/ +  y)  , 

i  >  0 

eward  per  unit  time 

in  state  i  is 

cifri  -  \±  f(i)  , 

'a, 

i  <  n 

0 

o. 

i  -  n 

0 

Also,  from  Equation  (3.8), 
n 

g(n)  -  l  0  (n)  q  , 
i-o  1  1 

where  the  Y  subscript  has  been  dropped  since  the  three  equations  for 
gain  rate  were  shown  to  be  equal  in  Theorem  3.1.  Naor  states  that  g(n) 
is  discretely  unimodal  in  n.  Thus,  if  Ag(n)  •  g(n)  -  g(n  -  1),  then  nQ 
such  that  Ag(nQ  +1)  <  0  -  Ag(nQ)  is  optimal.  Note  that 


n  n-1 

Ag(n)  -  l  0 . (n)q , (n)  -  J  0. (n  -  l)q,(n  -  1)  ,  (3.18) 

i-o  11  i-o  1  1 

where  q^(n)  denotes  the  expected  gain  rate  in  state  i  when  the  forced 
balking  point  is  n.  Thus,  q^n)  -  q^_^(n  -  1)  -  0  since  no  entry  is 
allowed  at  the  forced  balking  point. 

q^(n)  -  q^(n  -  1)  -  Af(i)  ,  for  i  -  0,  ...,  n  -  2; 

q  .  (n)  -  Af  (n  -  1) 

n-i 
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Substitution  for  q^(.)  in  Equation  (3.18)  yields 
n-2 

Ag(n)  -  l  {0,  (n)  -  0 .  (n  -  1)  >Xf  (i)  +  0  .(n)Xf(n  -  1) 
i-o  1  x  n 

Since  n  *  n  ,  f(i)  -  0.  Also,  0  , (n)  >  0  and  0 . (n)  -  0. (n  -  1)  <  0. 
o  s  n-l  i  i 

Thus,  Ag(n)  -  0  if 

.  n-2 

0  . (n)Xf (n  -  1)  -  l  {0 . (n  -  1)  -  0  (n)}Xf(i)  .(3.19) 

n-l  $..0  i  i 

If  p  *  X/y  i  1,  then  Equation  (3.19)  can  be  written  as 

(1  -  p)pn“lf(n  -  1)/(1  -  pn+1)  - 

ii"2 

l  {(i  -  pjpVd  -  pn)  -  (i  -  PjpVd  -  Pn+1)}fd)  . 

1-0  (3.20) 

After  formation  of  a  common  denominator  on  the  right-hand  side  and 

division  by  {(1  -  p) / (1  -  pn+*)}  >  0,  Equation  (3.20)  becomes 

>  _  n-2  . 

f(n  -  1)  -  {p(l  -  p)/(l  -  pn)}  l  plf(i) 

i-o 

ri^l  j  > 

Since  {(1  -  pn)/(l  -  p)}  -  l  p1,  this  leads  to  Ag(n)  -  0  if 

i-o 

f(n  -  1)  l  p1  -  p  l  p^d)  ,  n-2  .  (3.21) 

i-o  i-o 

Nov  consider  the  case  where  p  -  1.  If  p  -  1,  0,(n)  -  l/(n  +  1),  so 
Equation  (3.19)  becomes 

n**2 

f (n  -  l)/(n  +  1)  -  l  d/n  -  l/(n  +  1) }f (i)  . 

i-o 

After  some  algebra,  this  can  be  written  as 
s  n-2 

f(n  -  1)  -  {  l  f(i)}/n 

i-o 
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Since  this  is  the  limit  p  +  1  of  Equation  (3.21),  Equation  (3.21) 

can  be  used  for  all  values  of  p. 

Finding  the  range  of  X  over  which  n  ■  1  requires  finding  X 

o 

such  that  Ag(l)  ■  0,  but  Ag(2)  <  0.  If  g(l)  <  0  (R  <  C/U) ,  the 
system  is  trivial  since  no  customer  ever  enters.  Thus,  Ag(l)  -  0  for 
all  X  since  g(0)  ■  0.  For  Ag(2)  <  0,  Equation  (3.21)  is  a  linear 
function  of  X.  Finding  the  range  of  X  over  which  nQ  «  2  requires 
finding  X  such  that  Ag(?)  ■  0  but  Ag(3)  <  0.  Again,  for  Ag(2)  -  0, 
Equation  (3.21)  is  a  linear  function  of  X.  For  Ag(3)  <  0,  Equation 
(3.21)  is  a  quadratic  function  of  X.  This  pattern  continues  so  that 
finding  X  such  that  Ag(n  +  1)  -  0  or  Ag(n  +  1)  <0  requires  finding 
the  roots  of  an  n  degree  polynomial.  Use  of  Equation  (3.21)  is 
demonstrated  with  an  example. 

Suppose  the  reward  for  service  of  the  members  of  a  single  class 
of  customers  is  R  ■  5.  Also,  suppose  that  the  cost  per  unit  time  in 

the  system  is  C  -  2,  and  that  the  service  rate  of  the  single  server  is 

U  -  3.  {f (i) >  •  (R  -  C(i  +  l)/u)  -  {13/3,  11/3,  3,  7/3,  5/3,  1, 

1/3.  -2/3,  ...}  .  (Note  that  n  *7.)  First,  find  the  range  of 

values  of  X  for  which  nQ  -  1.  Ag(l)  •  0  for  all  X.  Ag(2)  <  0  if 

from  Equation  (3.21) 

1  .  0  . 

f(l)I  P  <  pj  P  f(i)  or  16.5  <  X 
i-o  i-o 

Thus,  n  -  1  if  X  >  16.5.  Now,  find  the  range  of  X  over  which  n  ■  2. 
'  o  o 

Ag(2)  ■  0  if  X  ■  16.5.  Ag(3)  <  0  if  from  Equation  (3.21) 

2  1 

f (2)  l  p1  >  p  l  p1f(i)  or  -2X2  -  12X  +  81  >  0  . 

i-o  i-o 

Solution  of  the  above  for  X  yields  Ag(3)  <  0  if  X  >  4.035.  Therefore, 
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n  ■  2  if  4.035  <  A  -  16.5.  Further  use  of  Equation  (3.21)  yields 
nQ  ■  3  if  2.1  <  A  ■  4.035.  Table  3.1  presents  the  results  of  the 
policy  iteration  program  of  Appendix  C  for  this  example.  These 
results  are  given  to  confirm  Equation  (3.21),  to  illustrate  Theorem 
3.2,  and  to  introduce  the  next  idea. 

TABLE  3.1 

Policy  Iteration  Results  for  Various  Values 
of  A  for  a  One  Class  Example  with  R  -  5, 

C  ■  2,  and  y  ■  3 


A 

no 

g 

0.1 

7 

0.431 

1.0 

5 

4.003 

2.1 

4 

6.944 

2.2 

3 

7.128 

4.02 

3 

8.993 

4.05 

2 

9.011 

16.4 

2 

10.998 

16.6 

1 

11.010 

100.00 

1 

12.621 

The  value  of  nQ,  the  forced  balking  point  for  the  social 
optimum  problem,  appears  to  decrease  as  the  arrival  rate  increases 
in  Table  3.1.  This  result  is  established  as  Theorem  3.3. 
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Theorem  3.3:  n  is  a  nonincreasing  function  of  X. 

-  -  —  o 

Proof:  Since  Naor  proved  that  g  is  discretely  unimodal 

in  n,  nQ  is  optimal  if  Ag(nQ  +  X)  <  0  “  Ag(nQ).  Aet 

n  (X)  denote  the  optimal  forced  balking  point  when  the 
0 

arrival  rate  is  X.  Let  n  ■  n^CX"),  where  X"  is  a  fixed 
value  of  X.  From  Equation  (3.21), 

g(n)  -  0  if 

n-1  n-2 

f(n  -  1)  l  P1  -  dl  PhD 
i-o  i-o 

for  n  «  2,  which  can  be  written  as 
Ag(n)  -  0  if 

f(n  -  1)(1  +  p  +  ...  +  Pn_1)  ■  pf(0)  +  P2f(l) 

+  ...  +  pn-1f (n  -  2)  . 

(3.22) 

As  X  increases,  p  increases.  Since  {f (i) )  decreases  as 

i  increases,  f (0)  >  f(l)  >  ...  >  f(n  -  2)  >  f(n  -  1). 

Thus,  as  X  increases,  the  right-hand  side  of  Equation  (3.22) 

increases  faster  than  the  left-hand  side.  This  eventually 

leads  to  Ag(n)  <  0  for  X  greatur  than  or  equal  to  some 

X'  >  X".  Therefore,  n  (X1)  <  n  (X")  .  Since  Ag(l)  >,  0  for 

o  o 

all  X,  continuation  of  the  above  argument  leads  to  the 
existence  of  X,M  such  that  nQ(X)  -  1  for  all  X  -  X'".  Thus, 
as  X  increases,  n  decreases  until  it  becomes  equal  to  oue. 
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Consider  again  the  value  of  g  as  X  increases  in  Table  3.1. 

Theorem  3.2  established  that  indeed  g  can  only  increase  as  X  increases. 

However,  note  that  g  appears  to  approach  a  limit  as  X  *+•  °°.  From 

*  Theorem  3.3,  nQ  decreases  as  X  increases,  so  that  eventually  n  «  1. 

If  n  ■  1, 
o 

0(1)  ■  U/(y  +  X)  and  0.(1)  «  X/ (y  +  X)  . 
o  1 

From  Equation  (3.8), 

n^ 

g(no)  -  l  0i(nQ)qi(no) 
i-o 

so 

g(l)  -  yXf (0)/(y  +  X) 

Dividing  numerator  and  denominator  by  X  and  then  letting  X  -»•  yields 


lim 


g(l) 


yf(o) 


For  this  example, 


lim 

X-k» 


g(l) 


13 


(3.23) 


3. 7  Tolls 

Lippman  and  Stidham  (1977)  define  an  optimal  congestion  toll  as 
"an  entrance  fee  that  induces  customers  acting  individually  to  behave 
in  a  socially  optimal  way."  Two  types  of  optimal  congestion  tolls'*" 
are  considered.  The  first  form  of  the  toll  is  analogous  to  the  tolls 
of  Naor  (1969)  and  Yechiali  (1971).  The  second  form  illustrates  what 
Lippman  and  Stidham  (1977)  call  the  monotonicity  of  a  toll  that  is  a 
function  of  the  number  of  customers  in  the  system. 

"*"The  "tolls"  developed  here  are  actually  payments  for  not 
joining  but  have  the  same  effect  as  charges  made  for  joining. 


li.,1 
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Suppose  that  the  assumptions  of  Model  I  are  modified  so  that  an 

additional  term,  Q ,  is  Introduced  into  the  cost  structure ,  where  0  is 

a  fixed  payment  to  any  customer  who  does  not  join  the  queue.  If  each 

customer  is  allowed  to  decide  whither  or  not  to  join  the  queue,  :hen 

each  will  maximize  his  own  expected  net  benefit.  Expected  net  benefit. 

for  joining  is  still  R  -  (i  +  l)C/y  if  i  customers  are  in  the  system, 

but  is  now  Q  for  not  joining.  Thus,  joining  serves  the  customer's 

self-interest  if  R  «*  (i  4-  ■  Q.  This  leads  to  an  M/M/l/n' 

s 

queueing  system,  where  n'  is  such  that 

S 

R  -  (n*  +  1)C/U  <  Q  -  R  -  n'C/ji 

»  s 

Thus, 

-  [<R  “  Q)y/c  ] 

which  id  just  Equation  (3.1)  with  R  replaced  by  R  -  Q. 

If  an  administrator  decides  who  joins  the  queue  and  he  wants  to 

impoue  a  limit  of  nQ  from  the  original  cost  structure  of  Model  I,  he 

can  do  so  by  changing  the  cost  structure  to  Include  Q.  Since 

(R  -  Q)y/C  “  Ry/C,  n'  ■  n  .  The  administrator  needs  to  choose  Q 

a  s 

such  that  n'  •  n  .  Thus,  he  must  find  Q  such  that 
so  H 

R  -  r.(nQ  +  l)/y  <  Q  -  R  -  CnQ/y  (3.24) 

?hlch  corresponds  tw  the  tollB  of  Naor  (1969)  and  Yechiali  (1971). 

Suppose  now  that  the  assumptions  of  Model  I  are  modified  so 
that  again  a  term  is  added  to  the  cost  structure.  Any  customer  who 
does  not  join  the  queue  is  paid  q*i,  where  q  is  a  fixed  amount  of 
money  and  1  is  the  number  of  customers  in  the  system.  Again,  assume 
that  each  customer  acts  to  maximize  his  own  expected  net  benefit.  The 
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expected  net  benefit  for  joining  is  still  R  -  (i  +  l)C/y;  however,  the 
expected  net  benefit  for  not  joining  is  q*i.  Joining  serves  the  self- 
interest  of  the  customers  if  R  -  (i  4-  l)C/y  ■  q*i.  This  leads  to  an 
M/M/l/n"  queueing  system,  where 

9 

n"  -  [y(R  +  q)/(C  +  yq)J 

Again,  if  an  administrator  decides  who  joins  the  queue  and  he 

wants  to  impose  the  social  limit  from  the  cost  structure  of  Model  I, 

he  can  do  so  by  changing  the  cost  structure  to  include  q*i.  Since 

R  -  (i  +  l)C/y  -  R  -  (i  +  l)C/y  -  q*i,  n  -  n".  The  administrator 

s  s 

wants  to  find  q  such  that  n"  ■  n  .  Such  a  value  of  q  is  determined  by 
n  s  o 

{yR  -  C(nQ  +  l)}/(ynQ)  <  q  -  (yR  -  CnQ)/{y(no  -  1) } . (3 . 25) 

3.8  Conclusion 

The  simplest  model  of  a  controlled  queue  considered  is  Model  I. 
The  semi-Markov  decision  process  and  linear  programming  formulations  of 
the  model  that  Yechiali  (1971)  presents  can  easily  be  solved  for  nQ  by 
use  of  a  policy  iteration  algorithm  and  a  simplex  algorithm,  respec¬ 
tively.  Although  Model  I  is  not  a  particularly  realistic  representation 
of  a  system  such  as  an  airport  landing  queue,  it  serves  as  a  point  of 
departure  in  the  development  of  more  realistic  models. 


CHAPTER  IV 


THE  SEVERAL  CLASS  MODEL 

This  chapter  defines  Model  II  in  which  several  classes  of 
customers  are  considered.  The  formulations  of  Model  I  presented  by 
Naor  (1969)  and  Yechiali  (1971)  are  extended  to  Model  II.  The  form  of 
the  optimal  policy  is  shown  to  be  a  set  of  forced  balking  points,  one 
for  each  class  of  customers.  The  Naor  type  of  formulation  of  Model  II 
is  not  easy  to  solve;  however,  the  formulation  of  this  model  as  a 
semi-Markov  decision  process  following  Yechiali  (1971)  lends  itself  to 
solution  by  both  policy  iteration  and  linear  programming. 

4.1  Model  II 

This  model  allows  M  classes  of  customers.  The  arrivals  from 

each  class,  m,  form  a  Poisson  stream  with  mean  rate  \  ,  where 

m 

m  m  1,  2,  ....  M.  The  service  times  of  the  single  server  are  inde¬ 
pendent,  identically  distributed,  exponential  random  variables  with 
mean  1/y.  Like  Model  I,  Model  II  is  also  an  M/M/1  queueing  system  but 
now  the  customers  come  from  M  separate  groups  or  classes  with  varying 
costs  and  rewards.  A  member  of  class  m  receives  a  reward  for  service 
of  R  and  pays  C  per  unit  time  he  spends  in  the  system.  To  avoid 

XQ  III 

trivialities,  R^  is  required  to  be  greater  than  or  equal  to  C^/y  for 
all  m.  Again,  two  objective  functions  are  considered,  one  for  self- 
optlmizlng  customers  and  the  other,  the  one  of  primary  interest,  for 
customers  acting  "socially."  Each  arrival  or  the  administrator  is 
assumed  to  be  able  to  determine  the  state  of  the  system  at  the  time 


of  each  arrival,  where  again  the  state  of  the  system  is  the  number  of 
customers  in  the  system.  An  optimal  policy  is  sought  for  each  objec¬ 
tive  function;  that  is„  a  set  of  join  or  balk  decisions  is  sought  for 
each  class  for  each  possible  state  of  the  system  to  maximize  each 
objective  function. 

A  self-optimizing  customer  decides  whether  or  not  to  join  by 

choosing  the  larger  of  the  expected  net  benefit  for  joining  and  the 

expected  net  benefit  for  not  joining  (which  is  zero).  For  a  member  of 

class  m  arriving  to  find  i  customers  ahead  of  him,  the  expected  net 

benefit  of  joining  is  R  ■  (1  +  1)C  /y.  For  the  social  optimum 

m  m 

problem,  an  administrator  decides  whether  or  not  a  customer  of  class  m 

can  join  when  i  customers  are  ahead  of  him.  The  sum  of  the  expected  net 

benefits  per  unit  time  of  all  arrivals  of  all  classes  is  shown  in 

Section  4.5  to  be  maximized  by  a  policy  that  imposes  a  vector  n  of 

o 

forced  balking  points  on  the  customers,  n  "(n  ,n  ,...,n  ), 

o  ox  o2  oM 

where  members  of  class  m  are  allowed  to  join  if  the  state  of  the  system 

is  less  than  n  but  must  balk  otherwise, 
o 

m 

4.2  Individual  Optimum  Problem 

The  solution  of  the  individual  optimum  problem  is  investigated 

from  two  points  of  view.  The  first  development  is  an  extension  of  the 

Naor  approach  to  Model  I.  If  an  arrival  from  class  m  finds  i  customers 

ahead  of  him,  his  expected  net  benefit  for  joining  is  R  -  (i  +  1)C  /y. 

m  m 

His  expected  time  in  the  system  is  (1  +  l)/y  because: 

1)  his  expected  service  time  and  that  of  the  i  -  1 
waiting  customers  ahead  of  him  is  1/y,  and 
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2)  the  expected  remaining  service  time  of  the  customer 
in  service  is  1/y  since  the  service  time  distribution 
is  memoryless. 

Since  the  net  benefit  for  balking  is  zero,  joining  the  queue  is  in  the 

arrival's  self-interest  if  the  number  of  customers  in  the  system  is 

less  than  n  ,  where  n  is  such  that 
s  s 

m  m 

R  -  (n  +  1)C  /y  <  0  -  R  -n  C  /y 
m  s  m  m  sm 

m  m 

Thus,  self-optimizing  customers  of  class  m  determine  a  balking  point 

n  such  that 
s 

m 

n  -  [r  y/C  1  ,  m  -  1,  2,  ....  M  .  (4.1) 

s  ^  in  mJ 

m 

(The  brackets  indicate  the  greatest  integer  function.)  This  is 

analogous  to  Equation  (3.1)  for  the  single  class  model.  Thus,  for  the 

individual  optimum  problem,  the  M/M/1  queueing  system  becomes  a  finite 

capacity  system  with  the  capacity  given  by  n*,  where 

s 

n 

s 

m 

However,  members  of  class  m  treat  the  system  as  if  it  had  capacity  n  ; 

s 

m 

that  is,  they  balk  when  n  or  more  customers  are  in  the  system. 

m 

Equation  (4.1)  can  also  be  developed  by  extending  the  Yechiali 

approach  to  Model  I.  Since  customers  only  use  the  state  of  the  system 

to  decide  whether  or  not  to  join  the  system,^  only  the  subclass  of 

stationary  Markovian  policies,  C  ,  needs  to  be  examined  for  the  optimal 

s 

'"Using  the  state  of  the  system  is  the  best  a  customer  can  do 
since  the  transitions  are  memoryless  and  the  horizon  is  infinite. 


(4.2) 


n* 

s 


max 

m 
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policy.  Thus,  a  policy  P*eC  is  sought  such  that  P*  yields 

S 

“  l  Di®  !  ,  Kan\  -  (i  +  •  <4-3) 

s  i*o  m“l 

Expression  (4.3)  represents  the  maximum  expected  gain  per  customer  for 
self-optimizing  customers.  D^(k)  is  the  probability  that  action  k  is 
chosen  under  policy  P  when  the  state  of  the  system  is  i.  Action 
k  *  (k^,  k^,  ...,  k^)  accepts  class  m  if  k  ■  1  and  rejects  class  m 
if  k  “0.  A  policy  P  is  a  set  of  join  or  balk  decisions  for  each 

in 

class  for  every  possible  state  of  the  system.  The  notation  k  (i)  is 

m 

used  to  emphasize  that  the  join  or  balk  decision  for  class  m  is  a 

function  of  i,  the  state  of  the  system.  Again,  R  -  (i  +  1)C  /u  is 

m  m 

the  expected  net  benefit  to  an  arrival  from  class  m  for  joining  if  i 
customers  are  ahead  of  him.  Equation  (4.3)  can  be  maximized  by 
setting  D^(k)  -  1  for  all  i  and  all  k  and  setting 

'l,  if  R  -  (i  +  1)C  /y  »  0 

m  m 

0,  otherwise 

The  same  arguments  that  were  used  earlier  in  this  section  to  develop 

Equation  (4.1)  can  now  be  used  again  to  arrive  at  Equation  (4,1). 

k  (i)  can  then  be  found  from  Equation  (4.1)  by  setting 
m 

if  i  <  n 

s 

m 

otherwise,  for  m  ■  1,  2,  ...»  M 
4 .3  Social  Optimum  Problem 

Three  different  formulations  of  the  social  optimum  problem  are 
presented  in  this  section.  As  in  Model  I,  an  infinite  time  horizon 


k  (i)  - 

m 


i. 

10. 


km(i>  “ 
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without  discounting  is  considered.  The  units  of  the  objective  function 
for  the  social  optimum  problem  are  dollars  per  unit  time.  Thus,  the 
administrator  wants  to  maximize  the  sum  of  the  expected  net  benefits 
per  unit  time  of  all  arrivals.  The  formulation  that  extends  Naor 
(1969)  i.3  presented  first. 

In  Section  4.5,  the  optimal  policy  is  shown  to  specify  a  vector 

n  of  forced  balking  points,  n  ■  (n  ,n  ,...,n  ),  where  class  m 

o  0  °1  °2  °M 

joins  if  the  state  of  the  system  is  less  than  n  and  balks  otherwise. 

°m 

The  sum  of  the  expected  net  benefits  of  all  arrivals  per  unit  time  when 
the  forced  balking  points  given  by  n  are  chosen  is 

M  _  M 

g(n)  •  l  X'(n)R  -  £  C  L  (n)  •  (4.4) 

i-i m  m  £«i m  m 

Here,  X^(n)  and  L^Cn)  are,  respectively,  the  effective  arrival  rate  of 
class  m  and  the  contribution  of  class  m  to  the  expected  number  of 
customers  in  the  system  when  forced  balking  points  n  are  employed. 
Equation  (4.4)  is  Equation  (3.2)  extended  to  M  classes  of  customers. 
Analytic  solution  of  Equation  (4.4)  for  the  optimal  n  is  a  formidable 
task.  (See  Edelson  and  Hildebrand  (1975)  for  the  case  M  -  2.) 

Although  a  heuristic  search  procedure  could  be  used  to  find  a  good 
solution  to  Equation  (4.4),  this  will  not  be  done  here  since  the  next 
formulation  of  the  problem  readily  lends  itself  to  solution. 

Tiie  second  and  third  developments  formulate  the  social  optimum 
problem  as  a  semi-Markov  decision  process.  This  second  formulation 
extends  the  method  of  Yechiali  (1971)  for  Model  I.  Here,  the  state  of 
the  system  is  again  the  number  of  customers  in  the  system.  An  arrival 
who  joins  when  i  are  in  the  system  is  given  his  expected  net  benefit 


50 


for  joining,  R  -  (i  +  1)C  /y,  upon  entry  into  the  system.  The 
in  m 

decision  to  be  made  by  the  administrator  for  each  state  of  the  system 

is  which  classes  if  any  to  admit.  This  decision  is  represented  by 

k  ■  (k^,  k^,  ...»  k^) ,  where  class  m  is  admitted  if  k^  ■  1  and 

rejected  if  k  *0.  In  Section  4.5,  n  ,  the  forced  social  balking 
mo 

m 

point,  is  shown  to  be  less  than  or  equal  to  n  ,  the  self-optimizer's 

m 

balking  point,  for  each  class  m.  Thus,  n*  from  Equation  (4.2)  serves 

s 

as  a  bound  on  the  state  space  required  for  the  semi-Markov  formulation. 

A  policy  P*£C  ,  the  class  of  stationary  Markovian  policies,  is  sought 
s 

such  that  P*  yields 


max  g  max  s  p  _  p  M 

Pec  '  pec  l  Vk)  »i  l  ,  -  (i  +  1)c>>  • 

s  s  i*o  m-1 

(4.5) 

The  units  of  Equation  (4.5)  are  dollars  per  unit  time,  the  same  units 
as  Equation  (4.4).  Comparison  of  Equation  (4.5)  with  Equation  (4.3) 
yields  some  insight  into  the  difference  between  the  social  optimum 
problem  and  the  individual  optimum  problem  for  Model  II.  The  main 
difference  is  that  the  social  optimum  formulation  given  by  Equation  (4.5) 
includes  the  steady  state  probability  that  the  system  is  in  state  i, 

p 

0J,  when  policy  P  is  used.  This  information  allows  the  administrator 

to  anticipate  net  benefits  from  future  arrivals.  The  other  differences 

between  Equations  (4.5)  and  (4.3)  are  that  n*  bounds  the  state  space  in 

s 

Equation  (4.5)  and  the  inclusion  of  Xm  in  Equation  (4.5)  makes  its  units 
dollars  per  unit  time  rather  than  dollars  per  customer  as  in  Equation 


(4.3). 


Equation  (4.5)  represents  the  optimum  gain  rate  for  a  semi-Markov 


decision  process  with  a  finite  state  space  and  a  finite  policy  space. 
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M 

The  policy  space  is  finite  since,  at  most,  2  actions  are  possible  for 

each  Of  at  most  n*  +  1  states.  Derman  (1962)  shows  that  a  nonrandomized 

P  — 

rule,  P€C^,  Is  optimal  for  such  a  process.  Thus,  D^(k)  can  be  dropped 
from  Equation  (4.5)  since  it  will  be  one  for  the  action  given  by  the 
optimal  k(i)  and  zero  oth'*wise.  In  other  words,  k(i)  is  the  only 
action  chosen  in  state  i  under  the  optimal  policy.  Equation  (4.5)  then 
becomes 


max 

PeC  AC.  8P 
s  a 


max 

Pec  n  c 

s 


k  (i)X  {R  -  (i  +  1) C/y} 
m  mm  m 

(4.6) 


Solution  of  Equation  (4.6)  by  policy  Iteration  and  linear  programming 
is  examined  in  Sections  4.6  and  4.7. 

The  third  and  final  formulation  of  the  social  optimum  problem 
for  Model  II  also  formulates  it  as  a  semi-Markov  decision  process  and 
extends  the  third  form  of  the  social  optimum  problem  presented  in 
Chapter  III.  The  difference  between  this  model  and  that  which  extends 
Yechiali's  formulation  is  that  now  a  joining  customer  receives  his 
reward  for  service  upon  entering  the  system  but  pays  his  costs  per 
unit  time  in  the  system  throughout  his  stay  in  the  system.  Since  the 
social  optimum  problem  of  Model  II  uses  an  infinite  horizon  and  no 
discounting,  the  timing  of  the  payments  does  not  affect  the  gain  rate 
of  a  given  policy.  The  state  of  the  system  must  indicate  the  number 
of  each  class  of  customers  in  the  system  so  that  the  cost  rate  can  be 
computed  at  all  times.  That  is,  the  charges  for  a  unit  of  time  with 
three  customers  of  class  one  and  two  of  class  two  in  the  system  are 
different  than  those  for  a  unit  of  time  with  two  of  class  one  and  three 
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of  class  two.  That  still  more  Information  is  needed  is  demonstrated 
by  the  following  example. 

Let  the  number  of  classes  be  two  and  the  state  space  be  (i,j), 
where  (i,J)  indicates  that  i  customers  of  class  one  and  j  customers  of 
class  two  are  in  the  system.  If  an  arrival  from  class  one  occurs  and 
the  customer  joins,  the  state  becomes  (i  +  l,j);  conversely,  if  an 
arrival  from  class  two  occurs  and  the  customer  joins,  the  state 
becomes  (i,j  +  1).  However,  if  a  service  occurs,  the  state  of  the 
system  is  unknown  since  the  state  space  does  not  indicate  the  class  of 
the  customer  in  service.  If  the  state  space  is  amended  to  Include  the 
class  of  the  customer  in  service,  it  becomes  (i,j,k),  where  k  *  1  if  a 
class  one  customer  is  in  service  and  k  ■  2  if  a  class  two  customer  is 
in  service.  This  definition  of  the  state  space  still  does  not  contain 
enough  information  to  keep  track  of  the  number  of  each  class  of 
customers  in  the  system.  For  Instance,  suppose  that  the  state  of  the 
system  is  (2,2,1).  Thus,  two  customers  of  each  class  are  in  the  system 
and  one  of  the  class  one  customers  is  in  service.  If  a  service  occurs, 
the  state  becomes  (1,2,?).  Since  the  service  discipline  is  first  come 
first  served  and  the  state  space  does  not  indicate  which  customer  is 
first  in  line,  the  class  of  the  customer  moving  into  service  is  unknown. 
The  system  occupation  costs  can  be  calculated  for  the  present  state, 
but  if  another  service  occurs,  the  number  of  each  class  of  customers 
in  the  system  is  unknown.  Thus,  the  state  space  must  be  expanded 
further.  In  fact,  the  state  space  must  indicate  exactly  what  the  queue 
looks  like.  A  state  space  like  (m^,  m^,  . . . ,  m^ ,  ...)  is  needed  where 
nij  indicates  the  class  of  the  customer  in  position  j.  For  example, 
state  (1,2, 1,1, 2)  indicates  that  a  customer  of  class  one  is  in  service 
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followed  by  a  customer  of  class  two,  two  of  class  one,  and  finally  a 
class  two  customer.  Although  this  state  space  becomes  large  and 
complicated  when  the  number  of  customers  in  the  system  is  large,  the 
social  optimum  problem  for  Model  II  can  still  be  formulated  in  this 
manner . 

Again,  only  the  class  of  stationary  Markovian  policies  needs  to 

be  considered.  For  each  state  of  the  system,  an  action  k  - 

(k^,  k^,  ...»  k^)  is  sought  to  maximize  the  gain  rate  of  the  system. 

As  before,  class  m  is  admitted  if  k  >1  and  must  balk  if  k  »  0. 

m  m 

Let  m  -  (m^,  m,, ,  , i.,  ...)  indicate  the  state  of  the  system,  where 

m.  gives  the  class  of  the  customer  in  position  j.  Also,  let  cf  (m)  be 
J  “ 

the  number  of  customers  of  class  m  present  in  state  m.  Note  that  only 

the  number  of  customers  of  each  class  in  the  system  is  required  to 

compute  the  occupation  costs  at  any  time,  but  the  position  of  the 

customers  in  the  system  is  required  to  keep  track  of  how  these  numbers 

change  as  the  system  goes  through  various  transition,  n*  from 

s 

Equation  (4.2)  still  serves  as  a  bound  on  the  number  of  customers 
allowed  in  the  system.  If  m'  denotes  the  set  of  all  m  satisfying 
M  _ 

£  a  (m)  -  n*  ,  (4.7) 

m-1  m  3 

the  objective  function  for  this  third  formulation  of  the  social 
optimum  problem  is 


max 

Pec 


gp 


max 

PEC 


I 


DP(k)  0P  l  {k  (m)A  R  -  C  (m)C  } 
—  —  m  mm  m  m 

m  m  m«l 


mem'  (4.8) 

Since  this  is  again  a  finite  state-space,  finite  policy-space  problem, 
a  deterministic  policy  is  optimal  according  to  the  results  of  Derman 
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P  — 

(1962)  and  thus,  D_  (k)  can  be  eliminated  from  the  formulation, 
m 

Equation  (4.8)  then  becomes 


max 

pec  nc.  sp 

s  a 


max 


M 


T  T 

PeC  AC,  1 •  L  , 

s  d  —  — ,  m  m«l 
mem 


{k  (m)A  R 
m  mm 


a (m)C J 
m  m 

(4.9) 


The  units  of  Equation  (4.9)  are  dollars  per  unit  time.  This  formula¬ 
tion  can  be  solved  using  either  policy  iteration  or  linear  programming. 
However,  since  all  three  formulations  are  shown  to  be  equivalent  in 
the  next  section,  this  formulation  will  later  be  dropped  because  it  is 
more  difficult  to  solve  than  the  previous  one. 


4.4  Equivalence  of  the  Three  Formulations 

The  equivalence  of  the  two  formulations  of  the  individual 
optimum  problem  has  already  been  shown  in  Section  4.2.  This  section 
establishes  the  equivalence  of  the  three  formulations  of  the  social 
optimum  problem  given  in  Section  4.3.  As  such,  it  extends  the 
equivalence  demonstrated  in  Section  3.4  for  the  one  class  model.  For 
convenience,  only  two  classes  of  customers  are  considered,  although 
the  method  used  generalizes  to  any  finite  number  of  classes.  The 
expected  gain  rate,  g  ,  for  the  Naor-type  extension  is,  from  Equation 
(4.4), 

2  _  2 

g«<n)  "  l  X,_(n)R  -  l  C  L  (n) 

n  *  .  m  m  ‘■■.mm 

m**l  m*l 

g^(n)  denotes  the  gain  rate  for  the  Yechiali-type  extension  and  g^(n) 
denotes  the  gain  rate  for  the  third  formulation.  (Forced  balking 
points  given  by  n  *  (n^,  n^)  correspond  to  control  limits  n^  -  1  and 
n£  -  1.)  g^(n)  and  g^(n)  are  developed  using  the  notation  of  Howard 

(1971)  and  both  are  shown  to  be  equal  to  g^(n). 
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gT(n)  is  developed  first.  Let  y^  be  the  cost  per  unit  time  in 
state  (i.j),1  where  (i,j)  indicates  that  i  customers  of  class  one  and 
j  of  class  two  are  in  the  system.  Thus, 


yiJ  '  iCl+JC2 

Let  b.  be  the  reward  for  a  transition  from  state  (i,j)  to  state 

ij  » l  j 

Here, 


R1  • 

if  i'  -  i  +  1,  3’ 

-  J 

r2  . 

if  i’  -  i,  j'  -  j 

+  1 

0  , 

otherwise 

Let  be  the  expected  holding  time  in  state  (i,j)  given  the  next 

transition  is  to  state  (i',j').  T^.  denotes  the  unconditional  expected 

waiting  time  in  state  (i,j).  Let  A^(i,j)  be  the  mean  arrival  rate  of 
class  m  customers  when  the  state  of  the  system  is  (i,j).  Here, 

X  ,  if  i  +  1  <  n 

X  (i,J)  -  '  a  °m 

lo  ,  otherwise 


With  this  definition,  T  can  be  found  from 

ij 

2 

1/(1  A  <±,j)>  ,  i  +  j  -  0 

m-1  m 

-  1 

T  * 

ij  2 

i/{y  +  l  x  <i,j)}  ,  i  +  j  ^  o 

m-1 

Also,  the  probability  of  a  transition  from  state  (i,j)  to  state  (i',j') 
is 


_  xFor  notational  convenience,  all  states,  m,  with  a^(m)  *  i  and 
a^Oa)  ■  j  are  combined  into  state  (i,j). 
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XA±,i)/il  X  ,  i  +  j  -  0;  i*  -  1,  j'  -  0 

1  m«l 


X2(i,j)/{I  Vi,j)}  »  i  +  j  -  0;  i'  -  0,  j'  -  1 

m*l 


x1(i,j)/{y  +  I  xm(i,j)} 

m“l 


i  +  j  >  0;  i*  -  i  +  1,  j’  -  j 


x9(i,j)/{y  +  i  Xo(i,j)}  , 


m-1 


i  +  j  >  0;  i'  -  i,  j'  -  j  +  1 


y/{y  +  l  X  (ij)}  ,  i  +  j  >  0;  i'  +  j'  -  i  +  j  -  1 

m 


m-1 


otherwise 


is  the  expected  reward  per  occupancy  of  state 

rij  "  [fjl  Vij.i'J'  +  bU  f> 

"  ’  yij  Pij,i'j'Tij,i’j’  + 

“  '  7ijTij  +  Pij ,(i+l)j  R1  +  Pij ,i(j+l)  R2 

Let  q_^.  denote  the  expected  reward  per  unit  time  in  state  ( 1 ,  j )  • 

qU  ‘  V?ij  ’  '  yU  +  {Plj,(i+Dj  R1  +  Plj.i<J+l)V/7lJ 

■  *’  (iC^  +  jC^)  +  X^(i,j)  +  X2  ( i  *  J )  ^ 


From  Howard  (I960),  gT(n)  can  be  written  as 


gT(«)  ■  I 


i+j<n* 


Vn)  qij 


(4.10) 


*.  1 


where  0^(n)  1®  th®  steady  state  probability  that  i  customers  of  class 
oae  and  j  customers  of  class  two  are  in  the  system  given  that  the 
forced  balking  points  are  given  by  n.  Thus, 

g^n)  -  l  0tj(n){-  (iCx  +  JC2)  +  (i.j)!^  +  X2(i,j)R2} 

i+j-n* 

s 


-  -  I  <  0i.(n)(lC 

i+j-n*  13 


l 

i+j<n 


Vn) 


+  X,R,  J  «,.(n)  .  (4.11) 

2  '*  i+j<n  13 
°2 

Equation  (4.11)  is  just  Equation  (4.9)  expressed  in  slightly  different 
notation.  Equation  (4.11)  can  be  rewritten  as 


Next,  gy(n)  is  shown  to  be  equivalent  to  g^(n) .  For  this 
formulation,  the  state  space  simply  gives  the  number  of  customers  in 
the  system.  The  cost  per  unit  time  in  state  i  is  y^  •  0.  The  reward 
for  a  transition  from  state  i  to  j  is 
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wejw*  *s\o  I  f?-y  -  >  **  ■■' 


^v^g»  '*'""•  *r*'wr« 


and 


i,i+l 


R,  -  (i  +■  l)C./li  with  probability  A  (i)/{t  A  (i)} 
1  A  1  ^-1  m 


R,  -  (i  +  1)C  /y  with  probability  A  (i)/{j  A  (i)} 


m-1 


m 


bi,j  "  °»  J  **  1  +  1 


Let  T.  ,  be  the  expected  holding  time  in  state  i  given  that  the  next 
* 

transition  is  to  state  j .  t  denotes  the  unconditional  expected 

waiting  time  in  state  i.  Let  A  (i)  be  the  mean  arrival  rate  of  class  m 

m 

when  the  state  of  the  system  is  i. 


A  (i)  - 

m 


A  , 

m 


0  , 


if  i  <  n 


m 


otherwise 


is  given  by 


1/li  A  (i) }  ,  i  -  0 


l/{y  +  l  A  (i)},  i  +  0 

m-1  ® 


The  probability  of  a  transition  from  state  i  to  j  is 


1  , 
2 


i  -  r,  j  «  l 


l  Am(i )/{y  +  l  Am(i)}  ,  i  +  0,  j  -  i  +  1 
m-1  m-1 


y/{y  +  l  A  (I)}  , 

m 


m-1 


i*0,  j-i-1 


otherwise 
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Thus,  r^,  the  expected  reward  per  occupancy  of  state  i  is 


ri  ”  ^  Pi,jbiJ  “  Pi,i+lbi,i+l 


The  expected  reward  per  unit  time  in  state  i  is  given  by 


qi  “  ri/Ti  *  (Pi,i+lbi,i+l)/Ti 


-  -  (i  +  D^/y} 


+  X2(i){R2  -  (i  +  l)C2/y> 


Assume,  without  loss  of  generality,  that  n  ■  n  .  Again,  from 

°1  °2 

Howard  (1960) , 


«  (n)  -  l  0(d)  q 

1  i«o  1  1 


n  -1 
°1 

l  0  (n)X  {r  -  (i  +  l)C  /y) 
i-o  1  1  x 

n  -1 
°2  • 

+  I  0 . (n)X  {r2  -  (i  +  l)C2/y> 
i-o 


n  -1  n  -1 

°1  _  °2 

-  X  R  l  0  (n)  +  X2R2  l  0±( 
i-o  i-o 


n  -1 
°2 

-C ,1  (X  /y) (i  +  1)0. (n) 

1  i-o  1 

n  -1 
°1 

-C ,1  (X  /y) (i  +  1)0. (n) 
H-n  1 
°2 
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n  -1 


-Cl  (X2Aj)(1  +  l)01(n) 
^i-o 


+  X’(n)R2 


n  -1 


-  {X1/(Xl  +  X2)}C1  l  {(Xx  +  X2)/y}(i  +  l)0i(n) 


n  -1 
°2 


-  {X2/(X1  +  X2)}C  l  {(X  +  X  )/y>(i  +  1)0  (n) 

i-o 


n  -1 
°1 

-C  l  (X  /y)  (i  +  1)0  (n) 
x  i-n  1  1 

°2 


(A. 12) 


n  -1 

°1  > 

The  termt-C-T  (X./y)(i  +  1)0.  (n))  only  appears  If  n  -  n  +  1. 
i-n  11  °1  °2 

°2 


For  a  finite  capacity  M/M/1  queue,  0  (n)  ■  (X  /y  )0  (n)  if 

2  1+1  1  1t1  1 


0  »  i<n*;  here,  X.  ■  Y  k  <i)X  ,  y...  =  y,  and  n*  «  “ax{n  }.  Thus, 
o  1  _«  a  to  i+i  o  mo 


m-1 


n  -1 
°2 


g^n)  -  l  X^(n)Rm  -  {Xj/^  +  X^}^  £  (i  +  l)0±+1(n) 

DJ-1  i*o 


n  -1 
°2 


-  U  /(X  +  X2)}C  l  (i  +  1)0  +1(n) 

"  i*o 


n  -1 
°1 


-  °i  L  «  +  i>»1+i(n) 
°2 


(A. 13) 


n  -1 


£  (i  +  1)0  (n)  ■  L(n),  the  expected  number  in  the  system.  Thus, 

i-o 
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n  -1  n  -1 

°2  °1 

{;x  /(X1  +  X  )>  l  <i  +  1)0  (n)  +  l  (i  +  1)0  (n) 

i-o  i-n  r 

°2 

Is  the  contribution  of  class  one  to  L (n)  and 

n  -1 
°2 

(X2/(X  +  X  )}  l  (1  +  1)0  (n) 

i«o 


is  the  contribution  of  class  two  to  L(n).  Finally, 


Vn>  ’  l  /i(n)Rm  '  ClLl(n)  ‘  C2C2(a) 

m-1 

2  _  2 
■  I  “  l  CL(n) 

m-1  m  m  o-l  m  “ 


This  establishes 


Theorem  4.1:  g^n)  *  ^(n)  -  gT(n) 


Since  the  three  formulations  of  the  social  optimum  problem  are 
equivalent,  the  easiest  to  solve  formulation,  g^Cn)  with  its  simple 
state  space  and  the  availability  of  policy  iteration  and  linear 
programming  solution  techniques,  is  chosen  for  use  in  this  paper. 

The  solution  techniques  for  this  problem  are  presented  in  Sections 
4.6  and  4.7. 


4.5  The  Form  of  the  Optimal  Solution 

For  the  social  optimum  problem  with  one  class  of  customers, 
Yechiali  (1971)  proves  that  a  control-limit  policy  is  optimal.  Also, 
both  Naor  (1969)  and  Yechiali  (1971)  prove  that  the  forced  balking 
point,  n  ,  for  the  social  optimum  problem  is  less  than  or  equal  to  the 
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balking  point,  n  ,  for  the  individual  optimum  problem.  The  purpose  of 
s 

this  section  is  to  prove  that  these  two  properties  of  the  optimal 
solution  carry  over  to  Model  II.  The  following  lemma  is  established 
first  and  then  used  to  prove  that  a  control-limit  policy  is  optimal  for 
several  classes  of  customers, 

Lemma  4.1:  For  the  social  optimum  problem,  reneging  is 
not  optimal. 

Proof:  Suppose  customer  A  of  class  m  arrives  at  time  T^ 
and  joins  the  system.  Later,  at  time  T^,  customer  A 
departs  the  system  before  he  is  served.  The  actions  of 
customer  A  affect  no  customer  who  arrived  before  him. 

The  contribution  of  customer  A  to  the  net  gain  of  the 
system  over  the  interval  (T^,  T^)  is  ”Cm(TD  -  T^)  <  0, 
his  holding  cost  for  the  time  he  is  in  the  system.  If 
no  other  customer  arrives  before  customer  A  departs,  his 
actions  have  no  effect  on  those  customers  arriving  after 
him.  If  other  customers  arrive  during  (T. ,  T  ) ,  the  only 

A  U 

effect  of  customer  A's  temporary  presence  in  the  system 
is  to  possibly  keep  a  profitable  customer  from  being 
allowed  to  join  the  system.  Thus,  the  contribution  of 
customer  A  to  the  system  is  negative  when  compared  with 
his  not  joining  the  system  at  all. 

Theorem  4.2:  A  control-limit  policy  is  optimal  for  each  class. 

Proof:  Suppose  that  there  exists  a  class  m  and  a  state  i 

such  that  in  the  optimal  policy  k  (i)  ■  1,  but  k  (i  -  1)  *0; 

m  m 
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that  is,  the  optimal  policy  for  class  m  is  not  a  control- 

limit  policy.  The  optimal  gain  rate  is  denoted  by  g*. 

If  0^  *  0,  then  a  policy  which  is  the  same  as  the  optimal 

except  that  k  (i)  **  0  also  yields  g*.  The  case  0.  4  0  is 

considered  next.  Consider  the  following  modification  to 

Model  II.  At  the  completion  of  a  service,  each  customer 

returns  his  expected  net  benefit  plus  his  occupation  costs 

for  the  service  just  completed.  The  customers  remain  in 

the  order  they  arrived,  but  the  administrator  recomputes 

the  expected  net  benefit  of  each  based  on  the  number  now 

ahead  of  him  and  uses  the  given  policy  to  decide  whether 

or  not  the  customers  can  stay  in  the  system.  Because 

there  is  no  discounting,  the  gain  rate  of  this  system  is 

equivalent  to  that  of  Model  II  for  the  same  policy.  Since 

the  transitions  are  Markovian,  all  relevant  information 

about  the  future  of  the  system  is  contained  in  the  current 

state  of  the  system.  (The  state  of  the  system  must  give 

the  position  of  every  customer  as  in  the  third  formulation.) 

A  customer  of  class  m  who  had  joined  when  i  customers  were 

ahead  of  him  would  be  forced  to  depart  when  the  number  of 

customers  ahead  of  him  dropped  to  i  -  1,  since  k  (i  -  1)  -  0. 

m 

Thus,  the  customer  would  be  forced  to  renege.  By  Lemma  4.1, 
the  given  policy  cannot  be  optimal  for  the  modified  system 
or  for  Model  II. 

The  next  theorem  establishes  the  relationship  between  the 
balking  point  for  a  class  in  the  individual  optimum  problem  and 


the  forced  balking  point  for  the  same  class  in  the  social  optimum 
problem. 


Theorem  4.3:  For  each  class  m,  n  ■  n 

-  o  s 

m  m 

Proof:  First,  the  decision  made  by  the  administrator  for 


a  given  arrival  does  not  affect  the  times  of  arrival  of 


any  customer  yet  to  arrive.  As  previously  shown,  n 

s 

m 

satisfies 

R  -  (n  +  1)C  /y  <  o  -  R  "  n  C  /u  .  (4.14) 

ms  m  m  s  m 

m  m 


Suppose  a  customer,  customer  A,  of  class  m  arrives  to  find 

the  state  of  the  system  i  *  n  .  Let  be  the  time  of 

m 

arrival  of  customer  A,  and  T  be  the  time  of  arrival  of 

B 

the  next  customer,  customer  B.  In  view  of  Equation  (4.14), 

let  a  <  0  be  the  expected  net  benefit  of  customer  A  joining 

the  system.  If  all  expected  costs  and  rewards  are  assigned 

to  a  customer  upon  arrival ,  the  contribution  of  the  interval 

(Ta,  Tg)  to  the  expected  net  gain  is  a  <  0  if  customer  A  is 

allowed  to  join.  The  decision  regarding  customer  A  does 

not  affect  the  expected  net  gain  of  customers  who  arrived 

before  him,  but  it  does  affect  the  expected  net  gain  of 

those  arriving  after  him.  If  customer  A  does  not  join,  the 

state  of  the  system  found  by  customer  B  and  all  others 

after  him  is  less  than  or  equal  to  the  state  of  the  system 

if  customer  A  joins.  Since  f  (i)  *  R  -  (i  +  1)C  /y,  the 

m  m  m 

expected  gain  for  a  customer  of  class  m  joining  when  i  are 
in  the  system,  is  a  strictly  decreasing  function  of  i,  the 
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contribution  to  the  net  expected  gain  of  the  interval 

(T_,  T  )  for  any  T-,  >  T  is  at  least  as  large  when 
B  C  C*  B 

customer  A  balks  as  when  he  joins.  Thus,  if  i  ■  n  , 

s 

m 

forcing  a  customer  of  class  m  to  balk  yields  a  larger 
net  expected  gain  or  gain  rate  than  allowing  him  to 
join. 

Theorems  4.2  and  4.3  suggest  a  simple-minded  approach  to  finding 

the  optimal  policy.  Since  a  control-limit  policy  is  optimal  for  each 

class  and  since  the  control  limit  is  bounded  by  n  -1  for  each  class  m, 

s 

m 

a  multidimensional  search  technique  can  be  used  to  find  the  optimal 

M 

policy  from  among  the  II  (n  )  possible  policies.  In  each  of  the  next 

m-1  sm 

two  sections,  a  more  efficient  solution  technique  than  the  suggested 
multidimensional  search  is  described. 

4.6  Policy  Iteration  Solution  of  the  Social  Optimum  Problem 

Equation  (4.6)  represents  the  optimum  gain  rate  for  a  continuous 
time,  infinite  horizon,  undiscounted,  semi -Markov  decision  process  with 
a  finite  state  space.  The  policy  iteration  algorithm  of  Howard  (1971) 
(see  Algorithm  C.l  in  Appendix  C)  will  be  applied  to  a  small  two  class 
problem  to  illustrate  the  method.  A  semi -Markov  decision  process  is  a 
semi -Markov  process  over  which  a  decision  maker,  here,  the  administra¬ 
tor,  has  some  control.  The  control  arises  from  the  ability  at  each 
transition  to  change  the  probability  distribution  for  the  next  transi¬ 
tion  of  the  process.  Here,  the  control  is  carried  out  by  choosing  the 
classes  of  customers  that  will  be  admitted  to  the  system.  Ignoring, 

for  the  moment,  the  form  of  the  optimal  policy  demonstrated  in  the 

M 

previous  section,  the  administrator  must  choose  from  among  2 
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alternatives  at  each  transition.  However,  since  the  transitions  are 
memory less  and  since  a  steady-state  exists  and  is  of  interest,  the 
decision  as  to  which  classes  to  admit  only  depends  on  the  number  of 
customers  in  the  system.  The  expected  net  benefit  rate  of  the  system 
is  influenced  in  two  ways  by  these  decisions: 

1)  The  expected  net  benefit  of  the  next  transition  depends 
on  the  classes  of  customers  allowed  to  join  the  queue. 

2)  The  expected  time  before  the  next  transition  depends 
on  the  classes  of  customers  allowed  to  join  the  queue. 

A  solution  to  a  two  class  example  Is  now  obtained  by  policy 
iteration  to  illustrate  the  use  of  the  technique.  In  this  example, 
class  one  customers  receive  a  reward  for  service  of  a  3  and  pay 
■  4  for  each  unit  of  time  spent  in  the  system.  The  arrival  rate 
for  class  one  is  X^  **  2  customers  per  unit  time.  Class  two  customers 
receive  ■  2  for  service  and  pay  C2  ■  3  for  each  unit  of  time  spent 
in  the  system.  The  arrival  rate  of  class  two  customers  is  "  4 
customers  per  unit  time.  The  single  exponential  server  has  a  service 
rate  capability  of  y  ■  4  customers  per  unit  time. 

From  Equation  (4.1),  n  “3  and  n  "2.  Thus,  from  Equation 

S1  S2 

(4.2),  n*  ■  3.  The  set  of  all  possible  actions  or  alternatives  con- 
s 

sists  of  {k:  k  ■  (0,0),  (1,0),  (0,1),  or  (1,1)}.  To  avoid  a  trivial 
system,  k(0)  #  (0,0);  that  is,  if  the  system  empties  out,  customers 


must  be  allowed  back  into  it.  Also,  Theorem  4.3  implies  that 
k(3)  -  (0,0)  and  k(2)  -  (k^O). 

P(0,0)  is  the  matrix  of  transition  probabilities  for  alternative 
k  -  (0,0). 
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P(0,0)  -  0 

1 

From  2 

State  “ 

3 


The  dashes  indicate  that  action  k  -  (0,0)  cannot  be  chosen  when  the 
state  of  the  system  is  0.  For  alternative  (0,0),  the  next  transition 
is  sure  to  be  the  completion  of  a  service.  Transition  matrices  for  the 
other  alternatives  follow. 

To  State 
0  12  3 

P(1,0)  -  0 

From  1 

State  2 

3 


To  State 


P(0,1)  »  0 

From  1 

State 

2 

3 


'i 

\ 


0  10  0 
0.67  0  0.33  0 

0  0.67  0  0.33 


To  State 
0  12  3 
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To  State 

0  12  3 

P(l,l)  -  0 

1 

From  „ 

State 

3 

To  see  where  the  entries  come  from,  consider  the  row  corresponding  to 
state  1  of  P(1,0).  The  transition  rate  from  state  1  to  2  is  given  by 
X  »  2.  The  transition  rate  from  state  1  to  state  0  is  given  by  the 
service  rate  U  -  4.  Thus,  the  total  rate  out  of  state  1  is  6  and  the 
rate  from  state  1  to  state  2  provides  0.33  of  the  total.  Thus,  the 
probability  that  the  competing  rates  yield  a  transition  from  state  1 
to  state  2  is  0.33.  Similarly,  the  probability  that  the  competing 
rates  yield  a  transition  from  state  1  to  state  0  is  0.67. 

Let  b(0,0)  be  the  matrix  of  expected  rewards  for  the  various 
transitions  under  alternative  k  *  (0,0). 

To  State 

0  12  3 

b(0,0)  -  0 

From  ^ 

State  2 

3 

Alternative  (0,0)  yields  no  benefits  since  no  customers  are  allowed  to 
join  the  system.  The  expected  rewards  for  the  various  transitions  for 


the  other  alternatives  follow. 
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b(l,0) 

From 

State 


To  State 
0  12  3 

' 

0  0  2  0  0 

10  0  10 

2  0  0  0  0 


b (0 ,1) 

From 

State 


To  State 
0  12  3 
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For  this  example,  the  expected  reward  per  transition  vectors  are 


r( 0,0)  -  -  ,  r (1,0)  -  2  ,  r(0,l)  -  1.25  ,  r(l,l)  -  1.5 


The  components  of  the  vector  of  expected  rewards  per  unit  time 
in  state  i  under  alternative  k  are  found  from 

q±(k)  -  ri(k)/ri(k) 

A 

Since  x(k),  the  vector  of  expected  waiting  times  under  action  k  is 
required,  it  is  given  below  for  the  various  alternatives. 


T(0,0)  - 


,  x (1,0)  -  f  0.5 


0.167 


0.167 


x(0,l)  «■  0.25  ,  x(l,l)  -  0.167 

0.125  0.1 


Thus,  the  expected  gain  per  unit  time  vectors  are 


q«M) 


,  q(l,0)  -  4  ,  q(0,l)  -  5  ,  q(l,l)  -  9 


A  aequanee  of  conmtanta,  {v^},  for  i  -  0,  n*,  referred  to 

by  Howard  (1971)  a*  the  relative  valuo  of  the  ays tern  starting  out  In 
atata  1,  la  required  to  utart  the  policy  Iteration  algorithm.  Au  he 
auiigaf'to,  tulw  v  “  0  for  all  i.  Thaaa  valuav  era  than  used  in  the 

X 

policy  da termination  portion  of  the  algorithm  which  i»  given  by 
liquation  (4.19). 

lfor  1  "  0,  1,  . . . ,  n*, 

nj 

uA(d)  -  “£X  (q1U)  +  {l/Tt(k)H>;“  l»1(J(k)vs  -  V,})  .  (4. IS) 

*1 

The  \aotov  dttO  givaa  the  policy  choaau  l’or  application  h  of  the 
ilgoi'ithm.  for  thin  example, 


d(l)  -  (1,1) 

(1,1) 

(1,0) 

(0,0) 

, 

In  onoi  atmuu,  tho  "elutiv't  value  of  » tuning  out;  in  «tat«  i,  v^,  uJIowh 
t;hu  adminUtvator  to  anticipatw  not;  hanefica  from  future  urrivaia  In 
that,  acute ,  If  all.  are  eat  equal  to  zero,  this  information  Ui  lout 
and  the  udminiat.ru tor  cun  do  no  butter  than  tho  e»olut. ton  to  ihvt 
individual  optimum  problem  which  U  given  by  dfl),  Aa  the  policy 


-..11 _ . _  J  ill’ll  -  Ihtt. 


M*iLh.,.x  .■...niAlhul 


iilfiUAu'CiJlMl 
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iteration  algorithm  is  applied  again  and  again,  better  knowledge  of  the 
v^'s  results  so  the  administrator  can  do  better  than  the  individual 
optimum  policy. 

The  gain  rate  and  relative  starting  values  associated  with  policy 
d(l)  are  then  computed  in  the  policy  evaluation  portion  of  the  algorithm 
given  by  Equation  (4.16). 

For  i  »  0 ,  1 ,  . . . ,  n* , 

s 


gr1(d1  (h)}+vi 


n* 

ri{d±<h»  +  I 


{<^00  }vj 


(4.16) 


In  Equation  (4.16),  g  is  the  expected  gain  rate  of  the  system  under 

policy  d(h).  Setting  v  ^  to  zero  reduces  the  number  of  unknowns  in 

us 

Equation  (4.16)  to  the  number  of  equations  and  yields  a  solution  vector 

v  whose  entries  are  values  relative  to  v  „  ■  0.  Solution  of  this 

n* 

_  s 

system  of  llnei  •  aquation  when  policy  7(1)  is  used  yields 


g  ■  2.55.’}  and  {v^}  m  {2.67,  1.60,  0.64,  0} 


This  new  set  of  v^'e  is  used  to  begin  the  second  application  or 
stage  of  "he  policy  iteration  algorithm.  For  this  new  sat,  Equation 
(4. 15^  yields 


7(2) 


(1,1) 

(1,0) 

(0.0) 

(0,0) 


Evaluation  of  policy  d(2)  givue 
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g  -  3.692  and  {v±}  -  (2.73,  1.85,  0.92,  0} 


This  new  sec  of  v^'s  leads  Co  termination  of  Che  algorlchm  because 

d(3)  -  d(2).  Since  Che  maximum  1  for  which  k, (1)  -  1  is  one,  n  ■  2. 

1  °x 

Similarly,  n  »  1. 

°2 


4.7  Linear  Programming  Solution  of  Che  Social  Opcimum  Problem 

The  formulation  of  Che  social  optimum  problem  given  by  Equation 

(4.6)  is  transformed  into  a  linear  programming  problem  in  this  section. 

As  in  Section  3.5,  the  approaches  of  Yechiali  (1971)  and  Fox  (1966)  are 

combined  to  yield  the  formulation  given  here.  The  linear  programming 

formulation  will  be  developed  first.  Then,  the  example  of  Section  4.6 

will  be  set  up  as  a  linear  program. 

Since  n  -  n*  for  all  m,  only  states  0  through  n*  can  have 
°m 

positive  steady  state  probabilities.  A  policy  P*e(T  is  sought  such  that 


(4.17) 


Equution  (A. 17)  is  Just  Equation  (4,6)  written  in  a  slightly  different 
form.  First,  rscull  that  in  Equation  (4.6),  D^(k)  was  dropped  because 
k  (i)  curried  the  same  Information.  Conversely,  k^d)  is  dropped  here 
in  favor  of  D^(k).  Second,  In  Equation  (4,6), 

1  1  » 

)  ... I  D  (k)  X  (R  -  (i  +  1)C  /u) 

Cj-o  cM-u  1  ■  « 


ia  Just  q^(k),  the  expected  gain  per  unit  time  in  state  i  when  action  k 
is  chosen  under  policy  l*  However,  q^(k)  ■  r^(k)/l^(k).  In  Equation 
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(4.17) ,  the  r^(k)  and  T^(k)  terms  are  summed  separately,  then  divided, 
as  opposed  to  Equation  (4.6),  where  q^(k)  is  formed  first.  Equation 

(4.17)  is  the  expected  gain  per  transition  divided  by  the  expected  time 
per  transition  which  is  indeed  the  expected  gain  per  unit  time.  The 
balance  equations  and  normalizing  equation  which  the  steady  state 
probabilities  {0jD^(k)}  must  satisfy  are 


n*  . 
_  s  1 


Ft  ...  I  0*  D*(k)  P  (k)  -l  ...  I  0X<k)  -  0  , 

>„  U  k-o  J  1  k  «o  k  -o  i  J 


PJP/r. 


for  j  -  0,  1 . n*  -  1 


(4.18) 


ll  ...  1  0?  D*(k)  -  1 

j-o  kj-o  kj^-o  J 


(4.19) 


where  0^  ■  0  and  D^(k)  -  0.  With  the  results  of  Fox  (1966),  as 
demonstrated  in  Section  3.5,  the  linear  program  with  fractional 
objective  function  given  by  Equations (4. 17)  through  (4.19)  is 
equivalent  to  the  following  llneur  program: 


n*  , 
a  1 


y  y  ...  y  y^io  r  o 

i«o  kj_-o  k^-o 


(4.20) 


subject  to 


»i  i 


I  1  •  *•  1  y i (k) ,00  -  ),  •  *  *  ),  yjd 

i-o  k^o  kM-o  1  A'J  ltj-o  UM-o  J 


(k)  -  0  , 


J  ■  0 . “  1 


(4.21) 


75 


and 

n*  i  1 

ll  ...  I  .  (k)f  (k)  -  1  ,  (A. 22) 

i*o  k  “o  k  “O 
1  M 

where 

yt(k)  -  0*  Dj(k)  -  0  .  (4.23) 

pit  _ 

Once  Equations  (A. 20)  to  (A. 23)  are  solved,  (k)  can  be  found 

from 

D^(k)  -  y  (k)/(  l  . ..  I  y, (k)}  .  (A.2A) 

k  ™o  k  -o 
1  M 

p*  _  _ , 

Fox  (1966)  ohows  that  D,  (k)  *  0  or  1  and  at  root  n*  +  1  of  the  D,  (k) 

i  si 

are  1.  Thus,  0^  can  be  found  from 

0**  “  yi(k)tt(k)  for  y^k)  >0  .  (A. 25) 

ft 

The  uncial  balking  point  for, each  clauu  m  can  be  found  from 

&0  -14'  max  Ci:  y^k^,  k2 . k^,  1>  . k^)  >  0}  , 

m  (A. 26) 

The  optimal  gain  rate  of  the  syotam  ia  given  by  the  maximum  value  of 
Equation  (A. 20).  Thus,  Equations  (A. 20)  to  (4,23)  cone ci tutu  the 
formulation  of  an  altwtnuta  solution  technique  to  policy  iteration  for 
the  social  optimum  problem  of  Model  XX. 

Hie  data  for  tha  example  problem  of  Section  A. 6  arc  repeated 
below  in  Table  4.1. 


I 
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TABLE  4.1 

Two  Class  Example  Problem 


Class 


R 


m 


m 


m 


1 

2 


3 

2 


4 

3 


2 

4 


The  single  server  provides  exponentially  distributed  service  times  at  a 
mean  rate  capability  of  four  customers  per  unit  time,  n*  was  found  to 
be  three.  P(k),  r(k),  and  T(k)  which  were  also  found  in  the  previous 
section  are  repeated  below  for  convenience. 


0 

1 

2 

3 

0 

1 

2 

3 

P(0,0)  - 

0 

t 

- 

- 

< 

O 

1 

> — *. 
o 

.H 

H. 

0 

1 

0 

0 

S 

1 

1 

0 

0 

0 

1 

0.67 

0 

0.33 

0 

2 

o 

l 

0 

0 

2 

0 

Or  67 

0 

0.33 

3 

l  0 

0 

1 

0 

3 

Mi 

«M 

~ 

J 

0 

1 

2 

3 

0 

1 

2 

3 

»T3 

S. 

o 

f 

0 

0 

1 

0 

0 

P(l,l)  «  0 

0 

1 

0 

o  1 

% 

1 

0.3 

0 

0.5 

o 

1 

0.4 

0 

0.6 

0 

) 

2 

«* 

M 

m 

Mt 

2 

w 

- 

mt 

- 

• 

3 

Mt 

Mi 

M 

*4 

i 

3 

mt 

- 

4 

r(Q,0)  - 

(-' 

1  H-" 

*•'(1,0) 

»  (  2 

\ 

|  r(0,X)  »  f  1.25  ^ 

|  vU 

.,1) 

“  I  1 

■»1 

0 

0 

0 


0.33 

0 


0 . 23 


0 .  A 
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A  A  A  A 


t(0,0)  - 

* 

i 

/ - s 

o 

r—< 

In 

’  0.5 

T  (0 ,1)  - 

'  0.25 

t(1,1)  - 

'  0.167' 

0.25 

0.167 

0.125 

0.1 

0.25 

0.167 

- 

0.25 

V. 

.  / 

.  - 

Using  this  information,  the  linear  programming  formulation  in  Equations 
(4.20)  to  (4.23)  for  the  example  problem  is 

max  2y0(l,0)  +  1.25yQ(0,l)  +  l,5y0(l,l)  +  d.33y  (1,0) 

+  0.25y^(0,l)  *5*  0.4y1(l,l) 

subject  to 

yl(°*°)  +  0'07yi(:Lf0)  +  O.Syj^CO,!)  +  0.4y^(l,l)  -  y0(l,0) 

-  y0(0,D  -  y0<l,l)  «  0 

y0(l,0)  +  y0(0,l)  +  y0(l.l>  +  y2(0,0)  +  0.$7y2(l,0) 

“  y^CO.O)  -  y^djO)  -  yn(0,X)  -  y0<l,l)  •*  0 

0.33yl(l,0)  +  0.5y1(0,l)  +  O.fiy^M)  +  y3(0,0) 

-  y2(0,0)  -  y2(i,0)  *  o 

0.3y0(l,0)  +  0,23y0(0,l)  +  0 ,167yQ(l*i)  +  Q.My^O.O) 

*  0.1677^(1.0)  +  0.l25yl(0,l)  +  U.ly^l.l) 

+  0.23y2(0,0)  +  0.167y2(l,0)  +  0.23y3(0,0>  «  l 

where  y^OO  *  0. 
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The  optimal  solution  of  this  linear  program  is  yg(l,l)  -  1.85, 
y^(l,0)  “  2.27,  y^CO.O)  -  0.92,  and  all  other  y^OO  -  0.0.  The 
maximum  value  of  the  objective  function  is  3.692.  From  Equation  (4.24), 
Dq*(1,1)  *  D**  (1,0)  -  d!J*(0,0)  -  1  and  all  other  D**(k)  -  0.0.  From 
Equation  (4.25),  0q*  -  0.308,  0^*  -  0.462,  and  -  0.23.  Finally, 

from  Equation  (4.26),  n  “2  and  n  “1. 

°1  °2 

4.8  Conclusion 

Model  I  has  been  extended  to  allow  for  several  classes  of 
customers.  The  characteristics  of  the  optimal  policy  are  found  to 
carry  over  to  this  extended  model,  Model  11.  A  control-limit  policy 
for  each  class  of  customers  maximizes  the  expected  net  benefits  per 
unit  time.  Compared  with  the  social  optimum,  self-optimizing  customers 
of  each  class  tend  to  overcongewt  the  system. 

Of  the  three  formulations  of  the  social  optimum  problem  that  are 
presented,  the  one  that  assigns  all  expected  coots  and  rowurds  to  u 
customer  at  the  time  of  his  entry  into  •  system  is  the  eisiast  to 
solve.  Two  solution  techniques,  policy  iteration  and  linear  programming, 
a  to  pr sweated  and  illuot rated, 


CHAPTER  V 


ERLANG  SERVICE  TIMES 

Now  that  the  restriction  on  the  number  of  classes  of  customers 

I 

has  been  eliminated  (see  Chapter  IV) ,  the  assumption  of  exponential 
service  times  which  is  not  very  satisfying  in  the  context  of  an  airport 
landing  queue  will  be  relaxed.  The  Erlang  density  which  allows  more 
flexibility  than  the  exponential  in  approximating  an  airport's  service 
time  density  is  introduced.  Model  III  extends  Model  II  to  Include 
Erlang  service  times,  Finally,  the  chapter  explores  two  methods  for 
solving  Model  III,  one  through  policy  iteration,  and  the  other  through 
mixed  integer  programming. 

5  •  1  Thu  Firlang  Density  Function 

The  following  introduction  to  the  Erlang  density  is  adapted  from 
Gross  and  Harris  (1974).  The  Erlang  density  function  la  a  subset:  of 
the  gamma  density.  Recall  that 

f(x)  ■  (l/{r(u)fia})  (xa  ^  exp(-x/R)}  for  a, 6  >  0  and  0  -  x  <  "» 

*  061  ,  i 

is  the  gamma  density  function.  !’(a)  “  /  xa  axp(-x)dx  and  «xp(.) 
i  o 

represents  e  raised  to  the  power  (.).  If  f(x)  is  the  density  function 

'  of  «  random  variable  X,  then  1S(X)  «  oiH  and  Var(X)  “  «{i  .  The  Erlang 

density  is  a  gamma  with  u  »  h  and  b  <*  l/(hn),  where  h  -  1,  2,  ...  and 

U  >  0  is  a  constant.  Thus,  If  t  has  an  Erlang  density,  then, 

f(»)  -  ll/(l»  -  l)!Hhu)'*Zh”L  oxp(-htiZ),  ?,  «  0 


Jj.-  -ik.‘iiw»J.,a.i:  t  jj,-, 


i, 
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Also,  E(Z)  -  1/y  and  Var(Z)  -  l/(hy2).  The  graphs  in  Figure  5.1 
demonstrate  the  increase  in  modeling  flexibility  of  the  Erlang  density 
over  the  exponential  density  function.  (Note  that  if  h  *  1,  the  Erlang 
density  is  an  exponential  density.)  The  effect  on  the  Erlang  distribu¬ 
tion  of  increasing  the  value  of  h  while  holding  the  mean  constant  is  to 
reduce  the  variance  of  the  distribution. 

To  extend  Model  II  in  a  straightforward  manner  requires  a  service 
time  distribution  with  the  Markovian  or  memory less  property.  Of  the 
continuous  distributions,  only  the  exponential  is  memoryless,  so  the 
usefulness  of  the  Erlang  distribution  may  at  first  appear  doubtful. 
However,  the  moment  generating  function  of  the  sum  of  h  independent, 
identically  distributed,  exponential  random  variables  with  parameter  0 
is  {0/(0  -  t)}*\  This  can  be  written  as  {hy/(hy  -  t)}^1  which  is  the 
moment  generating  function  of  an  Erlang  random  variable  Z  with 
E(Z)  -  1/y  and  Var(Z)  »  l/(hy  ).  Thus,  an  Erlang  random  variable  Z 
with  parameters  h  and  y  can  be  generated  by  the  sum  of  h  independent, 
identically  distributed,  exponential  random  variables  each  with  meat) 
1/hy.  The  technique  used  to  introduce  Erlang  nervine  times  whi'la 
maintaining  the  memoryleaa  property  la  to  artificially  break  a  service 
up  into  h  independent,  identically  distributed ,  exponential  phases. 

Of  course,  only  one  customer  is  allowed  in  service  at  any  one  tine. 

The  phases  are  imposed  for  the  convenience  of  rduthomui.lc.al  tructabillty 
and  do  not  necessarily  reflect  any  attributes  of  the  uctual  service 
process.  To  accommodate  Erlang  service  times,  the  state  space  of  the 
model  is  expanded  to  Indicate  the  number  of  phases  of  service  in  the 
system.  If  t:hu  system  contains  one  customer,  the  state  of  the  system 


Ca  .. 


.UA 


m  Miieii 


can  be  1  to  h.  If  the  system  contains  two  customers,  the  state  can  be 
h  +  1  to  2h.  In  general,  if  the  system  contains  i  customers,  the  state 
can  be  (i  -  l)h  +  1  to  ih. 

5.2  One  Class  Example  Problem 

A  simple  problem  is  presented  prior  to  the  presentation  of 
Model  III  to  illustrate  a  potential  problem  that  may  arise  so  that  the 
problem  can  be  avoided.  Consider  a  single  class  of  customers  with  the 
reward  for  service  R  -  3.5,  the  cost  per  unit  time  in  the  system  C  -  4, 
the  arrival  rate  X  ■  2,  and  the  mean  service  rate  capability  y  -  4. 
Suppose  that  an  Erlang  2  distribution  provides  a  close  fit  to  the  actual 
service  time  distribution.  Then,  each  service  cun  be  assumed  to  consist 
of  2  independent  exponential  phases,  each  with  mean  time  l/(hy)  -  1/8. 

The  Individual  optimum  problem  for  this  example  is  examined 

first.  Each  customer's  expected  service  time  only  depends  on  the  mean 

service  rate  which,  in  turn,  does  not  depend  on  the  form  of  the  service 

time  distribution i  However,  the  calculation  used  previously  for  the 

expected  time  a  customer  spends  in  the  system  depends  on  the  memoryluss 

property  of  the  service  time  distribution.  Although  this  difficulty 

precludes  finding  the  bulking  point  for  Erlang  distributed  service 

times  in  terms  of  the  number  of  customers  in  the  systum  us  easily  us 

before,  the  balking  point  cun  be  found  in  terms  of  the  number  of  phases 

of  service  in  the  systum  since  the  length  of  each  phase  of  service  is 

un  exponential  random  variable.  Let  n^  be  the  individual  optimum 

bulking  point  in  tonus  of  number  of  phasai.  of  service  in  the  system. 

u'  is  such  that 
s 

K  -  <  +  h)G/(hh)  <0  -  K  -  (o'  -  1  +  h)C/(hu). 

W  u 


(3.1) 
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Thus, 

n'  -  [Rhu/C]  -  (h  -  1)  ,  (5.2) 

where  the  brackets  represent  the  greatest  integer  function.  For  this 

example,  n'  -  6.  This  policy  implies  that  a  self-optimizing  arrival 
s 

will  balk  if  he  finds  three  customers  in  the  system  and  the  customer 
in  service  is  in  the  first  phase  of  service.  If  three  were  in  the 
system  and  the  customer  in  service  were  in  the  second  phase  of  service, 
the  arrival  would  join  the  system.  Since  the  phases  of  service  are 
purely  artificial  and  indistinguishable,  this  policy  cannot  be 
implemented.  Thus,  while  the  Individual  optimum  solution  may  turn  out 
to  be  one  that  can  be  implemented,  the  possibility  exists  that  it  may 
not  be  implementable.  This  uncertainty  casts  doubt  upon  the  usefulness 
of  this  formulation  of  the  problem.  The  same  difficulty  can  occur  in 
the  solution  to  the  social  optimum  problum. 

Solution  of  the  example  for  thu  social  balking  point  in  terms  of 
phases  of  sorvicu  by  policy  iteration  or  linear  programming  luadu  to 
thu  forcud  balking  point  occurring  when  four  phauuu  of  survico  uro 
present  in  the  system.  Since  thu  sorvicu  time  distribution  is  Krlung 
two,  thu  forcud  balking  point  occurs  when  two  customers  uro  In  the 
system  and  the  customer  in  uurvicu  is  in  his  first  phase  of  sorvicu. 
Again,  since  the  phases  uro  not  actually  pvuuunt  in  thu  service  itself, 
this  policy,  though  optimal  for  the  model,  Is  unrealistic  and  impossible 
to  implement. 

This  example  emphasizes  the  fact  t:lmL  an  implementublu  policy 
must  be  one  that:  does  not  force  an  arrival,  or  t  hu  admln.liit.rai  or  to 
determine  thu  service  phase  of  thu  customur  in  uurvicu,  Recall,  that  a 


policy  is  a  set  of  join  or  balk  decisions  for  each  class  of  customers 
for  each  state  of  the  system.  The  sat  of  all  possible  policies,  the 
policy  space,  contains  policies  that  cannot  be  implemented.  The 
policy  space  cun  be  reduced  to  only  those  policies  that  choose  the 
same  action  (the  same  Join  or  balk  decision  for  the  classes)  for  all 
states  representing  the  presence  of  the  same  number  of  customers. 

Such  policies  are  implement  able  because  they  do  not  require  an  arrival 
or  the  administrator  to  identify  the  service  phase  of  the  customer  in 
service.  Implsmentable  policies  will  also  be  refervul  to  as  llnkeu 
policies  since  the  actions  chosen  for  all  status  representing  the 
presence  of  the  same  number  of  customers  must  be  the  name. 

3.3  Model  lit 

Modul  111  has  M  classes  of  customers.  Thu  urvivals  for  each 

I 

class  m  form  a  Poisson  stream  with  mean  rate  X  .  The  survicu  t Lines  of 

the  single  server  are  independent,  identically  distributed,  KHung 

2 

random  varlubluu  with  moan  i/|i  ami  variance  !/(hu  ),  Kadi  survive 
time  is  generated  by  the  sum  of  h  independent,  identically  distributed, 
exponential  random  variables,  each  with  mean  l/(h|i).  Thu  statu  apace 
for  Modal  III  lti  uxpnndud  to  indicate  thu  number  of  phases  of  wurvicu 
iu  thu  system.  Phusus  (1  -  1) h  +  1  to  ih  represent  thu  presence  of  i 

customers ,  where  1*1,2 .  Thu  following  cost  structure  is 

imposed  on  the  operation  of  tlio  queueing  sysLuiu 

1)  liach  customer  of  cLuss  m  that  is  served  ruoufvue  a 
vowaid  of  R  dollars, 

2)  liach  unit  of  timu  a  class  in  customer  spends  in  Lhu 


85 


» 

A 

4. 

4 


* 


To  avoid  triviality,  R  ■  C  /\i. 

m  ni 

Again,  for  this  model,  each  arriving  customer  is  given  the 

choice  of  joining  and  being  subjected  to  the  cost  structure  or  balking 

and  not  paying  or  receiving  any  money.  Both  the  individual  and  social 

optimum  problems  are  considered.  Since  only  the  state  of  the  system 

upon  arrival  is  used  to  decide  whether  or  not  to  join  the  system,^- 

only  the  class  of  stationary  Markovian  policies,  C  ,  is  examined  for 

s 

the  optimum.  To  avoid  coming  up  with  a  policy  that  cannot  be  imple¬ 
mented,  the  decisions  for  all  of  the  states  corresponding  to  the 
presence  of  the  same  number  of  customers  must  be  identical.  Therefore, 
as  in  Model  II,  let  D^k)  be  the  probability  that  decision  k  is  chosen 
under  policy  P  when  i  customers  are  in  the  system.  Decision 
k  ■  (k^,  k2>  ....  k^)  accepts  class  m  if  k^  *  1  and  rejects  class  m 

if  k  -  0. 
m 


5.4  The  Individual  Optimum  Problem 

The  solution  of  the  individual  optimum  problem  will  be  investi¬ 
gated  first.  The  method  of  phases  will  be  used  to  learn  as  much 


information  as  possible  about  the  solution  to  the  problem.  When  this 

method  runs  into  difficulty,  other  methods  will  be  used  to  finally 

determine  {n  }  for  m  -  1,  2,  ...,  M. 
s 

m 

First,  remove  the  restriction  that  all  decisions  made  for  the 


states  representing  a  single  customer  be  identical.  As  in  Section  5.2, 

n*  ,  the  optimal  individual  balking  point  for  class  m  in  terms  of 
m 


Again,  using  the  state  of  the  system  is  the  best  a  customer 
can  do  since  the  transitions  are  memoryless  and  the  horizon  is 
infinite. 


phases  is  given  by  Equation  (5.2)  as 


n's  -  [r  hy/cj  -  (h  -  1) 

m 

This  balking  point  in  terms  of  phases  may  translate  into  a  balking 

point  in  terms  of  customers  that  cannot  be  implemented.  If  this 

happens,;  n  ,  the  individual  balking  point  for  class  m  in  terms  of 
sm 

the  number  of  customers  in  the  system,  is  either  [(n'  +  h  -  l)/h] 

m 

or  [(n*  +  h  -  l)h]  +  1.  That  is,  n  is  either  the  last  complete 

s  s 

m  m 

customer  represented  by  n*  or  one  more  than  that.  This  is  really 

s 

m 

enough  information  if  the  individual  optimum  is  only  used  to  bound 

the  state  space  for  the  social  optimum  problem.  (In  Section  5.5,  n  , 

m 

the  social  balking  point  for  class  m  is  shown  to  be  less  than  or  equal 

to  n  .)  The  larger  number  of  customers,  [.(nf  4-  h  -l)/h]  +  1,  can 

Sm  sm 

serve  as  the  bound  for  each  class.  The  individual  balking  point  for 

each  class  can  be  found,  if  desired,  by  different  means. 

One  method  of  obtaining  the  individual  balking  points  is  an 

extension  of  the  approach  used  in  Naor  (1969)  for  Model  I.  A  self- 

optimizing  customer  will  join  the  system  if  his  expected  net  benefit 

for  joining  is  greater  than  zero  which  is  his  expected  net  benefit  for 

balking.  Since  R  ■  C  /y,  f(n'  +  h  -  l)/hl  ■  1,  so  there  must  be  at 
m  m  4  s  J 

m 

least  one  customer  in  the  system  for  an  arrival  to  consider  balking. 

For  each  customer  in  the  system  but  not  in  service,  the  expected 
service  time  is  1/y.  The  difficulty  with  Erlang  service  times  is  that 
the  expected  remaining  service  time  for  the  customer  in  service  is  not 
1/y  since  the  Erlang  distribution  is  not  memoryless.  Thus,  the  problem 
is  to  find  the  expected  remaining  service  time  of  the  customer  in 
service.  Since  the  arrivals  for  each  class  form  a  Poisson  stream,  an 
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arrival  is  equally  likely  to  occur  at  any  point  during  i  „erT;‘"/; 
Following  Kleinrock  (1975),  the  expected  remaining  service  time  found 
by  an  arbitrary  arrival  is 

E (Remaining  Service  Time)  -  E(RST)  -  (E2(S)  +  Var (S) }/{2E(S) }  , 

(5.3) 

where  S  is  a  random  variable  representing  the  complete  service  time  of 

a  customer.  For  Erlang  service  times  with  E(S)  -  1/y  and 
2 

Var(S)  ■  1/hy  ),  Equation  (5.3)  becomes 
E(RST)  -  (h  +  l)/(2hu) 

The  individual  balking  point  in  terms  of  customers  can  be  found  from 

R  -  C  {n  /y  +  (h  +  l)/(2hy)}  <  0  ^  R  -  C  {(n  -  l)/y 

m  ms  m  m  s 

m  m 

+  (h  +  l)/(2hy)}  .  (5.4) 

This  leads  to 

ng  -  +  (h  -  l)/(2h)]  ,  (5.5) 

m 

where  the  brackets  indicate  the  greatest  integer  function.  The  set  of 
individual  balking  points  determined  for  the  classes  by  Equation  (5.5) 
solves  the  individual  optimum  problem. 

Equation  (5.5)  can  also  be  developed  using  an  extension  of  the 
approach  of  Yechiali  (1971)  for  Model  I.  Since  each  customer  considers 
only  his  own  net  benefit  in  deciding  whether  or  not  to  join  the  system 
and  since  all  customers  of  a  given  class  face  the  same  cost  structure, 
each  member  of  a  class  makes  the  same  decision  when  a  given  number  of 


customers  is  in  the  system.  Thus,  a  policy  P*eC  is  sought  such  that 

s 

P*  yields 

»  p  _  M 

SJ?  1  D;(k)  l  k  (i){R  -  iC  /y  -  C  E(RST) } 

PCG  7  l  u  ,  m  m  m  m 

s  i*o  m-1 


+  D‘(k)  l  k  (0) (R  -  C/y) 
u  m  mm 


(5.6) 


The  units  of  Equation  (5.6)  are  dollars  per  customer  so  it  maximizes 

the  expected  gain  per  customer  for  self-optimizing  customers.  Equation 

(5.6)  is  analogous  to  Equation  (4.3)  in  the  extension  of  Yechiali’s 

P  — 

formulation  to  Model  II.  Again,  D^(k)  is  the  probability  that  decision 
k  is  chosen  under  policy  P  when  i  customers  are  in  the  system. 
km(i)  *  0  if  class  m  balks  when  i  customers  are  in  the  system,  and 
km(i)  ■  1  if  class  m  joins  when  i  are  in  the  system.  Equation  (5.6) 

p  —  — 

can  be  maximized  by  setting  D^k)  ■  1  for  all  i  and  all  k  and  setting 


k  (i) 
m 


if  R  -  iC  /y  -  C  E(RST)  -  0 
m  m  m 

otherwise,  for  m  -  1,  2,  ...»  M. 


The  arguments  in  the  previous  development  can  now  be  used  to  develop 


Equation  (5.5).  Then, 


*m(i)  “  { 

°  lo 


if  i  <  n 


otherwise,  for  m  *  1,  2,  . . . ,  M. 


An  interesting  twist  to  the  normal  pattern  of  development  used 

in  this  paper  provides  numerical  results  to  confirm  Equation  (5.5);  in 

particular,  to  demonstrate  that  ng  can  indeed  be  [(n*  +  h  -  l)/h]  +  1. 

m  m 

As  noted  in  Lippman  and  Stidham  (1977) ,  the  difference  between  the 


self-optimizer  and  the  social  optimizer  is  that  the  self-optimizer 
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fails  to  consider  the  decrease  in  benefits  to  later  arriving  customers 


caused  by  his  joining  the  queue.  This  effect  is  called  an  external 


economic  effect.  Although  Lippman  and  Stidham  investigate  systems 


like  the  M/M /a  system,  the  external  economic  effect  applies  equally 


well  to  Model  III.  As  X  0  for  all  m  ■  1,  2,  . . . ,  M,  the  effect  of 

m 

a  given  customer’s  decision  to  join  on  later  arriving  customers 


declines  since  the  expected  interarrival  time  between  customers  is 


large  compared  with  1/y,  the  expected  service  time.  Thus,  it  seems 


reasonable  to  assume  that  as  X  -*■  0  for  all  m  «  1,  2,  . . . ,  M,  n 

m  o 

<  m 

approaches  n  from  below  (the  next  section  establishes  n  *  n  for 
s  os 

m  mm 

all  m) .  The  twist  is  to  find  a  social  optimum  problem  for  which 

n^  “  £(n'  +  h  -  l)/h]  +  1,  which  implies  that  n^  is  also 

mm  m 

[<nf  +  h  -  l)/h]  +  1. 
s„ 
m 

Using  the  policy  iteration  method  (see  Section  5.5)  for  solving 


the  social  optimum  problem,  a  simple  example  was  solved  for  which 

n  ■  £(n'  +  h  -  l)/h]  +  1  for  some  m.  In  the  example,  a  single 

s  s 

m  m 

class  of  customers  is  given  a  reward  R  •  3.99  for  service  and  charged 


C  *  4  per  unit  time  spent  in  the  system.  The  service  time  distribution 


is  modeled  as  an  Erlang  2  distribution  with  each  phase  of  service 


having  a  mean  service  time  of  1/8.  The  arrival  rate  of  the  customers 


is  0.4.  From  Equation  (5.2), 


n’  -  [Rhy/c]  -  (h  -  1)  -  6. 

s 


Thus, 

[(n’  +  h  -  l)/h]  “  3 

s 

and 


[(n^  +  h  -  l)/h]  +  1 


-  4. 
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From  Equation  (5.5), 

ng  *  [pP./C  +  (h  -  l)/(2h)]  -  4  . 

The  policy  iteration  method  of  Section  5.5  applied  to  this  problem 
yields  nQ  ■  4  which  implies  that  ng  “  4  and  confirms  the  result  of 
Equation  (5.5). 


5 . 5  Social  Optimum  Using  Policy  Iteration 

A  policy  iteration  approach  to  the  social  optimum  problem  is 

considered  next.  With  a  feu  minor  modifications,  the  proof  of  Theorem 

4.3  can  be  adapted  to  establish  that  n  *  n  ,  for  all  m,  for  Model 

o  s 

^  mm 

III.  Thus,  a  bound  on  the  state  space  required  for  policy  iteration 

can  be  determined  from  Equation  (5.5)  through 


n*  -  max  {n  }  . 

s  m  sm 


(5.7) 


A  policy  P*EC  ,  the  class  of  stationary  Markovian  policies,  is  sought 

9 


such  that  P*  y:f  aids 


max  m  max 

PeC  8P  “  PeC 
s  s 


n*  M 

{  l  k  U)\JR  -  1CJV  -  Cm  £(RST): 

l  “  ,  m  mm  m  m 

'  i*l  m-1 

+  D0;B|I0  l  ,  km(0)VR»  -  S»/u>}  •  (5-8> 

m-1  ' 


The  units  of  Equation  (5.8)  are  dollars  per  unit  time.  As  in  Models  I 
and  II,  the  main  difference  between  the  formulation  of  the  individual 
and  social  optimum  problems  is  that  the  social  optimum  formulation 


ASince  the  optimal  policy  has  not  been  shown  to  be  a  control- 

limit  policy,  n  needs  to  be  redefined  as  n  ■  1  +  max{i:  k  (i)  •  1} , 
o  o  m 

n  will  still  be  referred  to  as  a  balking  po?nt. 
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% 


4 


p 

includes  information,  here,  0^  that  allows  the  administrator  to 

anticipate  net  benefits  from  customers  who  have  yet  to  arrive. 

According  to  Derman  (1962) ,  a  nonrandomized  rule  PeC,  maximizes 

a 

P  — 

Equation  (5.8).  Thus,  D^(k)  can  be  dropped  from  Equation  (5.8)  since 
it  will  be  one  for  the  decision  chosen  for  each  i  and  zero  otherwise. 
Equation  (5.8)  then  becomes 


max 

PCX  rtC,  SP 
s  d 


max 

PSC  fiC. 


k  (i)X  {r  -  iC  /V  -  C  E(RST)} 
m  m  m  m  m 


P  M  ) 

+  0n  l  k .  (0U  (R  -  C  /y) V.  (5.9) 

0  m-1  m  m  m  m  J 

It  should  be  obvious  that  if  Equation  (5.9)  works  for  Erlang  service 
times,  it  works  for  any  general  service  time  distribution  with  a  finite 
mean  and  variance  since  E(RST)  can  be  found  from  Equation  (5.3). 
However,  Equation  (5.9)  cannot  be  solved  by  the  techniques  at  hand 
since  in  the  formulation  in  Equation  (5.9),  the  problem  is  not 
represented  as  a  semi-Markov  (or  Markov)  decision  process.  To 
demonstrate  this,  consider  that  if  the  system  enters  state  i  from 
state  i  -  1,  an  arrival  has  occurred  and  T^,  the  expected  waiting  time 
in  state  i,  is  E(RST) ,  while  if  the  system  enters  state  i  from  state 
i  +  1,  a  service  occurred  and  the  expected  waiting  time  in  state  i  is 
1/y,  the  expected  service  time.  Unless  the  service  times  are  expon¬ 
entially  distributed,  E(RST)  1/y.  Thus,  another  approach  is  required 
to  solve  the  social  optimum  problem. 

The  inability  to  solve  Equation  (5.9)  using  the  techniques 
developed  leads  back  to  the  method  of  phases.  The  social  optimum 
problem  without  the  restriction  requiring  the  same  decision  be  made  for 
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every  state  representing  the  presence  of  the  same  number  of  customers 
can  be  written  as 


max 

PECs 


hns  M 

£  l  «  i 

s  j-o  J  m-1 


(j  +  h)C  / (hy) > 
m 

(5.10) 


P  — 

Again,  the  results  of  Derman  (1962)  imply  that  (k)  can  be  dropped 
from  the  formulation  since  a  nonrandomized  rule  is  optimal  so  that 
Equation  (5.10)  becomes 


max 

pec  nc 

s  c 


h  s  p  M 
max  y  0P  y 

Pec.nc,  £  i 

d  1  **o 


s 


m-1 


k  (j)X  {R 
m  mm 


(j  +  h)C  /(hy)} 
m 

(5.11) 


This  is  the  same  formulation  that  led  to  difficulties  in  Section  5.2; 
that  is,  the  optimal  policy  may  be  one  that  cannot  be  implemented 
since  it  may  require  arrivals  to  identify  the  service  phase  of  the 
customer  in  service. 

Since  linking  decisions  places  additional  constraints  on  the 
problem,  the  opLlr,a''  solution  to  the  "linked"  problem  yields  a  gain 
rate  that  is  no  greater  than  that  found  by  solving  Equation  (5.11). 

A  simple  way  to  find  a  good  linked  policy  is  to  use  the  following 
algorithm: 


Algorithm  5.1: 

a)  Use  policy  iteration  to  solve  Equation  (5.11)  for  g 

and  n’  for  all  m,  where  n’  is  the  optimal  forced 
o  o 

m  m 

balking  point  in  terms  of  phases  of  service  in  the 
system. 

b)  If  the  optimal  solution  car.  be  implemented,  stop.  If 

not,  go  to  Step  (c) . 


c)  For  each  class  whose  balking  point  occurs  at  the  last 

phase  of  a  customer's  service,  fix  n  »  f(n'  +  h  -  1) /hi  . 

oo  J 

m  m 

For  the  other  classes,  n  is  either  \ (n'  +  h  -  l)h] 

o  L  o 

m  m 

or  [(n*  +  h  -  l)/h]  +  1. 

m 

d)  Among  all  possible  sets  generated  from  Step  (c) ,  find 

the  set  {n  } ,  m  »  1,  2 ,  . . . ,  M,  of  balking  points  that 
o 

m 

yields  the  maximum  gain  rate. 

This  procedure  is  fairly  easy  to  implement,  although  it  may  require  use 

of  the  policy  evaluation  portion  of  the  policy  iteration  algorithm  on 
M 

2  policies  in  Step  (d) .  However,  it  may  not  find  the  optimal  solution 
since  it  searches  only  a  few  linked  policies  in  the  region  of  the  phase 
optimum.  In  fact,  the  few  policies  examined  are  control-limit  policies. 
(Although  the  proof  of  Theorem  4.2  does  not  directly  apply  to  Erlang 
service  times ,  it  seems  reasonable  to  assume  that*  the  optimal  policy 
will  again  be  a  control-limit  policy.)  A  more  thorough  search  of  the 
linked  policies  lends  itself  to  a  mixed  integer  programming  formulation. 

5.6  Social  Optimum  Using  Mixed  Integer  Programming 

This  section  extends  the  phases  of  service  formulation  of  Model 
III  so  that  all  possible  linked  policies,  policies  that  can  be  imple¬ 
mented,  are  examined  to  determine  the  optimal  policy.  Recall  that  a 
linked  policy  mikes  the  same  decision  for  all  states  that  represent  the 
presence  of  the  same  number  of  customers. 

Equation  (5.11)  represents  the  social  optimum  problem  in  terms 


of  phases  of  service  in  the  system  with  no  restriction  on  the  decisions 
made.  Equation  (5.11)  can  be  formulated  as  a  linear  program  similar  to 
that  developed  in  Chapter  IV  [see  Equations  (4.20)  to  (4.23)]. 
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hn*  .  . 

si  1 

““I  l  ‘”1  y,(k)r.  (k) 

i-o  k^-o  k^*o  ■'i  '  i 


subject  to 


hn5 1  i 


l  l  ■■■  l  yAW,  ,00  -  I  ■•■l  y,00  -  o  , 

i-o  k^o  k^-o  k^*o  k^*o  3 


hn*  1 
s  1 


I  I  I  yi(k)xi 

i»o  k  “o  k  *o 
1  M 


(k) 


yt(k)  -  0  . 


j  -  0,  1,  ....  (hn*  -  1) 

3 


(5.12) 


hn*  bounds  the  total  number  of  phases  of  service  required  for  the  social 

optimum  problem,  where  h  is  the  number  of  phases  per  service  and  n*  is 

s 

found  from  Equations  (5.5)  and  (5,7).  r_^(k)  is  the  expected  reward  per 

occupancy  of  state  i  when  decision  k  is  chosen.  T^(k)  is  the  expected 

waiting  time  in  state  i  if  action  k  is  chosen.  P  (k)  is  the  proba- 

i » J 

bility  of  a  transition  from  state  i  to  j  if  action  k  is  chosen.  y^(k) 

can  best  be  explained  by  observing  that  the  stationary  probability  of 

the  system  being  in  state  i  under  policy  P  is  0^  *  y^(k)x^(k)  for 

y^(k)  >  0.  Thus,  y^(k)  is  in  essence  a  weighted  probability  that  the 

system  is  in  state  i  under  policy  P.  Policy  P  is  a  set  of  decisions 

{k(i)}»  i  -  0,  1,  ...,  hn*. 

s 

The  only  remaining  task  is  to  include  in  the  formulation  the 
requirement  that  the  decision  for  each  class  be  the  same  for  the  states 
of  the  system  that  represent  the  presence  of  the  same  number  of 
customers.  In  general,  states  (i  -  l)h  +  1  to  ih  represent  the 
presence  of  i  customers,  where  i  -  1,  2,  ..., 


n*.  If  the  policy  is 


o  be  one  that  can  be  implemented,  the  decisions  k(j)  “  (k^(j), 

k2(j),  J ) }  made  for  states  j  »  (i  -  l)h  +  1  to  ih  must  be  the 

M 

same.  If  i  *  15  2,  n*  -  1,  2  actions  are  possible.  [The  number 

s 

of  actions  for  some  states  could  be  reduced  by  using  Equation  (5.5)  to 

set  k  (j)  *  0  for  all  j  -  (i  -  l)h  4-  1  to  ih,  where  i  »  n  .]  Thus, 
m  s 

m 

for  a  policy  to  be  implementable ,  the  actions  chosen  for  the  states 

*ore**f  ’.ting  each  of  i  ■  1,  2,  . . . ,  n*  -  1  customers  must  be  either 

s 

V  '  -  i)h  +  1}  -  k{(i  -  l)h  +  2}  -  ...  -  k(ih)  -  (0,0,. .. ,0) 

t. 

k{(i  -  j>h  +  1}  *  k{  (i  -  l)h  +  2}  -  ...  -  h(ih)  «  (1,0,. ..,0) 
or 


* 

¥ 

k 


k{(i  -  l)h  +  1}  «  k{(i  -  l)h  +  2}  -  ...  -  k(ih)  -  (1,1 . 1)  . 

These  either /or  constraints  do  not  fit  the  usual  form  of  linear 
programming  constraints.  The  method  suggested  in  Taha  (1971)  will  be 
used  to  convert  the  either/or  constraints  to  constraints  that  can  be 
used  in  a  mixed  integer  program. 

For  i  •  1,  2,  . ..,  n*  -  1,  let 

rl  if  k{ (i  -  l)h  +  1}  -  ...  -  k(ih)  -  (1,0,  . .  ,0) 

Z  (1,0,... ,0)  -  ■ 

0  otherwise. 

fl  if  k{ (i  -  l)h  +  1}-  ...  -  ic(ih)  -  (1,1,. ..,1) 


zid,i,...,i) 


0  otherwise. 
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Z^(0,0, . . . ,0)  is  not  needed  since  k  must  be  0  if  k  ^  (1,0,..., 0) , 
and  ...  ,  and  k  +  (1,1,..., 1).  The  (Z^(k)}  are  binary  variables  which 
indicate  whether  or  not  action  k  is  chosen  when  i  customers  are  in  the 
system.  If  B  is  a  large  positive  number,  the  either/or  constraints  are 
equivalent  to 


ih  < 

-BZ  (1,0,. ...0)  +  l  y  (1,0 . 0)  «  0 

j«(i-l)h+l  2 


—  < 

-B{1  -  Z  (1,0,. ...0)}  +  l  l  y  (k)  -  0 

j-(i-l)h+l  W(1,0 . 0)  2 


-BZ  (1,1, ... ,1)  +  1  y  (1,1, ... ,1)  -  0 

j«(i-l)h+l  2 


in  r  _ 

-B{1  -  Z  (1,1,. ..,1)}  +  l  i  y,(k)  -  0 

1  j»(i-l)h+l  k^(l, !,...,!)  2 


where  1*1,  2,  . . . ,  n*  -  1 • 

s 


(5.13) 


To  illustrate  the  equivalence  of  Equation  (5.13)  to  the  either/or 
constraints,  consider  that  if  action  (1,0,..., 0)  is  chosen  for  each 
state  from  (i  -  l)h  +  1  to  ih,  then  Z^(1,0, . . . ,0)  *  1  and  all  other 
Z±(k)  -  0.  Thus, 

ih 

-BZ. (1,0 , ... ,0)  +  l  y . (1,0, . . . ,0)  -  -B 

j«(i-l)h+l  2 

ih 

+  l  y . (i,o, . . . ,0)  -  o 

j»(i-l)h+l  3 


since  B  is  a  large  positive  number.  Since  y^ (k)  ■  0  and 


-B{l  -  Z1(1,0,...,0)>  -  0, 
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-b{i  -  z1(i,o,. 


ih 

,0)>+I  l 

j-(i-l)h+l 


y^Ck)  -  0 


only  if  all  y^ (k) r  k  j*(l,0,...,0)  and  j  «  (i  -  l)h  +1,  . ..,  ih,  are 
zero.  Since  action  (1,0, ...,0)  is  chosen  for  all  states  (i  -  l)h  +  1 
to  ih,  these  y^(k)  are  indeed  zero.  The  other  constraints  in  Equation 
(5.13)  are  also  satisfied.  Thus,  choosing  action  k  -  (1,0,...,0)  for 
all  states  representing  customer  i  satisfies  Equation  (5.13).  Similar 
arguments  can  be  made  f>r  any  other  action  chosen  for  all  the  states 
representing  a  single  customer.  If  different  actions  are  chosen  for 
the  states  representing  a  single  customer,  all  Z^(k)  are  zero  and  at 
least  one  of  the 

_  ih 

-BZ  (k)  +  l  y  (k)  -  0 

1  j-(i-l)h+l  2 

constraints  is  violated.  Thus,  only  those  policies  that  choose  the 
same  action  for  all  states  representing  a  single  customer  satisfy 
Equation  (5.13). 

Equation  (5.13)  allows  at  most  h  of  the  (y^(k)},  j  »  (i  -  l)h  +  1, 

—  —  P  —  —  P 

...,  ih;  all  k,  to  be  positive.  Since  y.(k)  »  0./x.(k)  and  0,  is  a 

J  1  5  1 

probability, 

y  (k)  -  i/^-a  {t  (k)})  . 

2  j  ,k  2 

Thus,  the  large  positive  number,  B,  required  in  Equation  (5.13)  is 
determined  by 

B  ■  k/ (k)  }) 


Equation  (5.13)  does  not  constrain  the  decision  when  zero  or  n* 
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t. 


( 


k 


customers  are  in  the  system.  When  n*  customers  are  in  the  system, 

action  k  *  (0,0, . . . ,0)  must  be  chosen  since  n  ■  c*  for  all  m.  Thus, 

°m 

k{(n*  -  l)h  +  1}  -  ...  -  k(n*h)  -  (0,0,..., 0)  .  (5.14) 

s  s 

Zero  customers  are  in  the  system  only  if  zero  phases  of  service  are  in 
the  system,  so  only  one  state  represents  the  condition  of  having  zero 
customers  in  the  system.  Thus,  the  problem  of  linking  decisions  does 
not  arise  when  zero  customers  are  in  the  system.  Equations  (5.12)  to 
(5.14)  constitute  a  mixed  integer  programming  formulation  of  Model  III. 
The  first  example  of  Section  5.2  will  be  formulated  as  a  mixed  integer 
program  to  illustrate  the  method. 

A  single  class  of  customers  receives  a  reward  R  ■  3.5  for  service 
and  is  charged  C  «  4  for  every  unit  of  time  spent  In  the  system.  The 

mean  arrival  rate  of  the  Poisson  stream  of  customers  is  X  *  2  customers 

per  unit  time.  The  mean  service,  rate  capability  of  the  single  server 
is  y  -  4  customers  per  unit  time.  Suppose  that  an  Erlang  two  distribu¬ 
tion  provides  a  reasonable  model,  of  the  service  time  distribution. 

From  Equations  (5.5)  and  (5.7), 

n*  -  ng  «  [UR/C  +  (h  -  l)/(2h)]  -  3 

Thus,  hn*  -  6  is  a  bound  on  the  number  of  phases  of  service  required 

for  the  social  optimum  problem.  The  mean  service  time  of  each  piase 
is  1/ (hp)  «  1/8.  The  transition  probabilities  are 

Pj ,j-i(°)  “  lf  j  “  1*  2’  6 


1 
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and 

C  .2, 

j  -  i  +  2 

0.8, 

j  -  i  -  1 

o  » 

otherwise,  i  »  1,  2,  ...,  4 

• 

(Recall  that  decision  0  cannot  be  cnosen  if  1  ■  0,  and  decision  1 
cannot  be  chosen  if  1  ■  5  or  6.)  Calculation  of  the  expected  reward 
per  occupancy  of  a  state  yields 

rj<0)  *  °»  *  *  lp  2*  6 

and 

r.U)  *  ...  (1){R  “  <3  +  h>C/(hy)};  j  -  0,  1,  ....  4 

J  j  tj+n 

The  expected  waiting  times  are 


T^CO)  -  0.125,  j  -  1,  2,  ....  6 


and 


T  (1)  - 


0.5,  j  -  0 

0.1,  j  -  1,  2,  3,  4 


Also,  B  -  b / (min{Xj (k) }}  -  2/0.1  -  20;  let  B 
tlon,  Equation  (5.12)  to  (5.14)  yield 


-  25.  With  this  informa- 


max  2.5yQ(l)  +  0.4yi(l)  +  0.3y2(l)  +  0.2y3(l)  +  0.1y4(l) 
subject  to 

y0(1)  '  yl(0>  “  "  0 

yx(0)  +  y^l)  -  y2(0)  -  0.8y2(l)  -  0 

y2(0)  +  y2(l)  -  y0(l)  -  y3(0)  -  0.8y3(l)  -  0 

y3<0)  +  y3(l)  -  0.2y^(l)  -  y4(0)  -  0.8y4(l)  -  0 

74(0)  +  y4(l)  -  0.2y2(l)  -  y5(0)  -  0 

y5(0)  -  0.2y3(l)  -  yfi(0)  -  0 
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G.5y0(l)  +  0.125y1(0)  +  O.ly^l)  +  0.125y2(0)  +  0.1y2(l) 

+  0.125y3(0)  +  0.1y3(l)  +  0.125y4(0)  +  O.ly^l) 

+  0.125y5(0)  +  0.125y6(0)  -  1 

-25Z1(1)  +  yi(l)  +  y2(l)  -  0 
-25{1  -  Z^l)}  +  yi(0)  +  y2(0)  -  0 
-25Z2(1)  +  y3(.l)  +  y4(l)  -  0 
-25(1  -  Z2(l)>  +  y3(0)  +  y4(0)  -  0 

^uere 

Zi(l),  Z2(l)  -  0  or  1;  y0(l)  ■  0,  y^ (k)  -  0  for  j  -  1,  ....  4; 
lc  -  0,1;  y5(0)  -  o,  y6(0)  -  o 

The  optimal  solution  to  this  problem  Is 

y0(l)  -  1.12,  y^l)  -  1.4,  y2(l)  -  1.75,  y3<0)  -  0.63, 
y4(0)  -  0.35,  Z^l)  -  1 

All  other  y^ (k)  and  Z2(l)  are  0.  The  optimal  gain  rate  is  g  -  3.895. 
This  solution  gives  an  optimal  social  balking  point  of  n  »  2. 


,  5.7  Conclusion 

,  In  this  chapter,  the  model  was  generalized  to  provide  an  Erlang 

service  distribution  for  the  server.  This  was  done  because  the  flex- 

* 

ibility  of  the  Erlang  distribution  will  be  needed  in  Chapter  VI  to 

> 

develop  a  realistic  model  of  the  actual  service  time  distribution  for 
an  airport  landing  system.  For  the  policy  iteration  solution  procedure, 
a  control-limit  solution  was  assumed;  however,  no  restrictions  were 
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placed  on  the  form  of  the  solution  found  by  mixed  integer  pro¬ 
gramming. 

The  policy  iteration  procedure  is  a  heuristic  algorithm  that 
ia  easy  to  implement.  On  the  other  hand,  the  mixed  integer 
programming  formulation  allows  identification  of  the  optimal  solution 
but  becomes  cumbersome  for  large  problems.  The  policy  iteration 
procedure  requires  the  inversion  of  a  matrix  that  has  as  many  rows 
and  columns  as  there  are  states.  The  mixed  Integer  program  has  an 
equality  constraint  for  each  state.  These  problems  may  hamper  the 
solution  of  problems  with  several  hundred  states  or  more;  however, 
the  user  of  these  methods  should  be  able  to  alleviate  such  problems 
by  taking  advantage  of  the  sparseness  of  the  matrix  (constraints). 
Since  the  policy  iteration  procedure  is  easy  to  implement,  the  user 
should  consider  trying  it  first.  If  its  results  are  unsatisfactory, 
the  user  can  then  try  mixed  Integer  programming. 


^TSjrr  stv 1 1-|. 


CHAPTER  VI 

AIRPORT  LANDING  QUEUE  APPLICATION 

In  this  chapter.  Models  II  and  III  at,*  applied  to  airport 
landing  queues.  Data  from  several  sources  are  used  to  develop  the 
parameters  of  the  models;  however,  the  particular  airport  that  is 
modeled  is  the  Greater  Pittsburgh  International  Airport.  The  work  of 
Adler  and  Naor  (1969)  is  used  to  help  refine  the  results  obtained  for 
the  Pittsburgh  Airport. 

6.1  Customer  Parameters 

The  purpose  of  this  chapter  is  to  determine  how  entry  to  the 
landing  queue  should  be  controlled  during  peak  traffic  periods  when  a 
single  runway  is  being  used  for  landings  only.  The  use  of  a  peak 
traffic  period  to  determine  the  control  policy  makes  the  benefits  of  a 
control  policy  apparent.  In  light  traffic,  little  or  no  control  is 
required.  Thus,  a  peak  traffic  period  gives  the  best  indication  of  an 
airport’s  capability  to  land  aircraft  effectively.  Only  commercial 
jet  aircraft  are  considered  in  this  study,  although  other  traffic  uses 
the  Pittsburgh  Airport.  The  other  traffic  generally  uses  another 
runway  during  periods  of  heavy  use  of  the  Pittsburgh  Airport  so  that 
the  models  can  be  applied  realistically  to  commercial  jets  only. 

Commercial  jet  aircraft  are  divided  into  the  following  five 
classes: 


i 


l 

■M 


jWi !  i  ■h>‘ hit '/  Ml  &  ‘■*:X 
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1)  Class  One  -  Three-engine,  wide  body 

2)  Class  Two  -  Four-engine,  wide  body 

3)  Class  Three  -  Three-engine,  regular  body 

4)  Class  Four  -  Four-engine,  regular  body 

5)  Class  Five  -  Two -engine 

The  aircraft  that  make  up  the  various  classes  are  listed  in  Table  6.1. 


TABLE  6.1 

Aircraft  Categorized  by  Class 


Class 

Manufacturer 

Aircraft 

1 

Lockheed 

L-1011 

McDonnel 1-Doug las 

DC-10 

2 

Boeing 

747 

3 

Boeing 

727 

4 

Boeing 

707 

McDonnell-Douglas 

DC-8 

5 

British  Aircraft  Corp. 

BAC-111 

McDonnell-Douglas 

DC-9 

The  mean  arrival  rate  of  each  class,  X  ,  is  determined  first. 
The  Transportation  System  Center  (1978)  lists  the  hourly  number  of 
landings  on  4  August  1978  for  the  Pittsburgh  Airport.  A  peak  number 
of  32  landings  occurred  from  1600-1700  hours  and  again  from  1900-2000 
hours.  Thus,  the  overall  mean  arrival  rate  used  in  this  study  is  32 
aircraft  per  hour.  Rather  than  find  class  arrival  rates  by  attempting 
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to  determine  which  aircraft  arrived  during  these  periods,  class  arrival 
rates  are  approximated  by  first  detei^aining  the  proportion  of  total 
yearly  passenger  jet  traffic  at  the  Pittsburgh  Airport  that  is  repre¬ 
sented  by  each  class.  The  oean  arrival  rate  of  each  class  during  a 
peak  period  is  approximated  by  multiplying  32  by  the  proportion  of 
yearly  jet  traffic  that  each  class  represents.  The  Civil  Aeronautics 
Board  and  Federal  Aviation  Administration  (1978)  list  che  total  number 
of  departures  (and  thus,  arrivals)  from  the  Pittsburgh  Airport  for  each 
type  of  aircraft  for  a  12-month  period  ending  30  June  1978.  The 
percentage  of  the  total  number  of  commercial  jet  departures  and  the 
computed  mean  arrival  rate  of  each  class  are  given  in  Table  6.2. 

TABLE  6.2 

Percentage  of  Yearly  Jet  Traffic  Represented 
by  Each  Class  and  Approximate  Mean  Arrival 
Rate  for  Each  Class  in  a  Busy  Period 


Class 

%  of  Yearly 

Jet  Traffic 

X  (Aircraft/Hr) 
m 

1 

2.1 

0.672 

2 

0.2 

0.064 

3 

26.2 

8.384 

4 

6.2 

1.984 

5 

65.3 

20.896 

Noah  et  al.  (1977)  provide  three  estimates  of  the  direct 
operating  cost  to  the  airlines  of  each  hour  of  flying  time  due  to 
delay.  The  costs  included  in  the  estimates  are  the  costs  of  fuel. 
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airline  crew,  aircraft  maintenance ,  and  depreciation  of  the  aircraft. 

The  estimates  derived  from  three  different  sources  [Civil  Aeronautics 

Board  (CAB)  (1975);  Reck  et  al.  (1975); and  Rogers  et  al.  (1975)]  are 

given  in  Table  6.3.  For  each  class  m,  C  is  approximated  by  averaging 

m 

the  three  estimates  given  in  Table  6.3.  These  approximations  are  shown 
in  Table  6.4. 

The  computation  of  R  ,  the  reward  for  the  service  of  an  aircraft 

m 

of  class  m,  is  rather  involved.  R  is  found  by  multiplying  the  average 

m 

profit  per  passenger  by  the  average  number  of  passengers  on  a  flight  of 
class  m.  The  average  profit  per  passenger  is  obtained  by  dividing  the 
sum  of  the  profits  of  all  domestic  airlines  for  a  year  by  the  number  of 
passengers  that  flew  in  that  year.  1974  figures  are  used  because  the 
estimates  of  costs  used  for  Cm  were  made  in  1975  and  thus  were  probably 
based  on  1974  data.  Moles  and  Wlmbush  (1976)  report  the  sum  of  the 
profits  of  all  domestic  airlines  in  1974  as  $799,298,000  and  list 
189,733,000  as  the  number  of  passengers  for  that  year.  Thus,  the 
average  profit  per  passenger  is  $4.21.  Moles  and  Wimbush  (1976)  also 
give  the  fraction  of  seating  that  was  occupied,  the  passenger  load 
factor,  as  0.555  for  1974.  The  average  number  of  passengers  on  a 
flight  of  class  m  is  the  average  capacity  of  class  m  aircraft  times 
the  load  factor.  The  average  capacity  of  aircraft  of  class  m  is 
determined  by  a  weighted  average  of  the  seating  capacities  of  the 
aircraft  that  make  up  the  class.  The  weight  for  a  given  type  of 
aircraft  in  class  m  is  the  proportion  of  yearly  flights  into  the 
Pittsburgh  Airport  by  class  m  aircraft  that  is  accounted  for  by  the 
particular  aircraft  type.  These  proportions  can  be  computed  for  the 
Pittsburgh  Airport  from  data  given  by  the  CAB  and  FAA  (1978).  The 
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TABLE  6.3 

Estimates  of  Direct  Operating  Cost  Per  Hour 
of  Flight  for  Each  Class  of  Aircraft 


Class 

CAB  ($/Hr) 

Reck  ($/Hr)  Rogers  ($/Hr) 

1 

1729.20 

1718.40 

1620.00 

2 

2415.60 

2295.60 

1980.00 

3 

871.20 

860.40 

780.00 

4 

1112.40 

1093.20 

1080.00 

5 

694.80 

671.40 

660.00 

TABLE  6.4 

Cost  Per  Hour  of  Flight  for  Each  Class 


Class 


C  ($/Hr) 
m 


1  1689.00 

2  2230.00 

3  837.00 

4  1095.00 


5 


675.00 
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maximum  seating  capacities  of  the  various  aircraft  In  the  classes  are 

taken  from  Aviation  Week  (1977).  The  weights  and  seating  capacities 

are  given  in  Table  6.5.  The  average  capacity  for  each  class  can  be 

found  by  summing  the  products  of  the  last  two  columns  of  Table  6.5. 

These  results  are  given  in  Table  6.6.  For  each  class  m,  R  is  the 

in 

average  profit  per  passenger  ($4.21)  times  the  load  factor  (0.555) 

times  the  average  capacity  of  an  aircraft  of  the  class  (Table  6.6). 

The  values  of  R  are  givv...  in  Table  6.7. 
m 

6.2  Server  Parameters 

The  first  task  that  must  be  accomplished  is  to  define  a  service. 
In  actual  practice,  the  control  of  an  aircraft  approaching  the 
Pittsburgh  Airport  begins  with  the  Cleveland  center,  is  then  trans¬ 
ferred  to  approach  control,  and  is  finally  passed  on  to  the  control 
tower.  Each  of  these  control  sectors  is  really  a  group  of  air  traffic 
controllers.  In  turn,  each  controller  may  handle  from  one  to  about 
six  aircraft.  During  busy  periods  at  the  Pittsburgh  Airport,  most 
commercial  jet  traffic  is  landed  on  Runway  28  center,  while  departures 
operate  from  Runway  28  right.  If  Runway  28  center  and  the  airspace 
near  it  are  defined  as  the  service  facility,  then  a  single  server 
model  is  appropriate.1  Since  FAA  rules  require  a  minimum  separation 
between  aircraft,  on  approach  and  on  the  runway,  the  service  time  of 
an  aircraft  is  defined  to  be  the  period  of  time  that  the  runway  is 
cleared  for  use  by  the  aircraft.  This  definition  is  easiest  to 

^If  an  airport  uses  several  runways  for  landings,  a  single 
server  model  could  still  be  used  for  a  given  runway  if  the  class 
arrival  rates  are  adjusted  so  they  are  class  arrival  rates  at  the 
particular  runway  rather  than  at  the  whole  airport. 
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TABLE  6.5 

Seating  Capacity  and  Within  Class  Weights 
for  the  Aircraft  in  Each  Class 


Class 

Aircraft  Type 

Seating  (maximum) 

Weight 

1 

DC-10 

380 

0.46 

L-1011 

400 

0.54 

2 

Boeing  747 

500 

1.00 

3 

Boeing  727-100 

131 

0.65 

Boeing  727-200 

189 

0.35 

4 

Boeing  707-100B 

181 

0.63 

Boeing  707-300,  300B, 
300C 

189 

0.20 

DC-8-20 

176 

0.01 

DC-8-50,  62 

189 

0.01 

DC-8-61 

259 

0.15 

5 

BAC-111 

89 

0.22 

DC-9-10 

90 

0.14 

DC-9-30 

115 

0.53 

DC-9-50 

139 

0.11 

. . . 
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TABLE  6.6 

Average  Seating  Capacity  of  Each  Class 


Class 

Average  Capacity 

1 

391 

2 

500 

3 

151 

4 

194 

5 

108 

TABLE  6.7 

Reward  Per  Flight  for 

Each  Class 

Class 

<$> 

m 

1 

914 

2 

1168 

3 

353 

4 

453 

5 

252 
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implement  when  the  airport  is  busy;  then,  the  service  time  of  Aircraft 
B  (which  follows  Aircraft  A)  is  defined  to  be  the  time  from  Aircraft  A's 
crossing  the  end  of  the  runway  (or  some  other  easily  defined  point)  to 
Aircraft  B's  crossing  the  same  point.  Thus,  the  service  time  distri¬ 
bution  is  obtained  from  data  taken  during  busy  periods  at  the  airport. 
This  service  time  distribution  is  assumed  to  apply  whether  or  not  the 
airport  is  busy. 

Data  were  taken  at  the  Pittsburgh  Airport  on  8  June  1979  under 
good  weather  conditions.  Observations  of  the  service  times  were  made 
at  a  radar  scope  similar  to  that  used  by  the  air  traffic  controllers. 

The  radar  scope  was  used  instead  of  direct  visual  observation  for  the 
following  reasons: 

1)  The  information  given  on  the  scope  included  the 
altitude  of  the  aircraft  which  provided  an  alternate 
definition  of  the  end  of  a  service;  in  this  case, 

1200  feet,  the  elevation  of  the  runway,  was  used 
rather  than  the  end  of  the  runway. 

2)  The  information  given  on  the  scope  included  the  type 
of  aircraft  so  that  service  time  observations  not 
involving  aircraft  in  the  five  classes  (or  aircraft 
with  similar  landing  speeds)  could  be  dropped. 

3)  The  pattern  of  aircraft  on  the  scope  faciiitated  the 
identification  of  busy  periods. 

The  service  times  which  were  found  using  a  decimal  minute  stopwatch 
are  given  in  Table  6.8. 

The  sample  mean  of  the  Pittsburgh  data  is  x  *  1.81  minutes  and 
the  sample  standard  deviation  is  s  *  0.413  minutes.  The  parameters  of 


TABJ.E  6.8 

Observed  Service  Times  (Pittsburgh) 


Observation 

Time 

(Min) 

24 

1.68 

25 

1.00 

26 

2.39 

27 

2.27 

28 

1.61 

29 

1.57 

30 

2.33 

31 

1.69 

32 

1.55 

33 

1.41 

34 

1.48 

35 

1.87 

36 

2.58 

37 

1.73 

38 

1.80 

39 

2.05 

40 

1.27 

41 

1.41 

42 

1.48 

43 

1.87 

44 

1.56 

45 

1.72 

Observation 


Observation 


Time 

(Min) 


1.34 
1.53 

2.34 
2.50 
1.94 
1.79 

1.32 
1.18 

2.57 
1.87 

1.93 
1.79 

1.94 

1.32 

1.57 
1.62 
1.63 

1.32 
1.41 
1.48 
1.16 
1.62 


112 


the  Erlang  distribution,  service  rate  y  and  Erlang  number  h,  are 
estimated  by  both  the  method  of  moments  and  the  method  of  maximum 
likelihood.  [See  White,  Schmidt,  and  Bennett  (1975).] 

A  A 

For  the  method  of  moments,  the  point  estimates, y  and  b,  of  the 
parameters  of  the  Erlang  distribution  are  found  using  Equations  (6.1) 
and  (6.2). 

A 


y  -  l/x 

(6.1) 

h  -  l/(s2  U2) 

(6.2) 

For  the  data  in  Table  6.8,  the  estimates  are  u  -  0.552  aircraft  per 

A 

minute  and  h  -  19.3.  Since  the  unit  of  time  used  in  the  calculation 
of  the  customer's  parameters  is  an  hour,  the  estimate  of  y  needs  to  be 
converted  to  y  *  33.15  aircraft  per  hour.  Since  h  has  to  be  an  integer, 
19.3  is  infeasible.  This  problem  will  be  resolved  shortly. 

The  point  estimates,  u  and  h,  of  the  parameters  of  the  Erlang 
distribution  using  the  method  of  maximum  likelihood  are  found  using 
Equations  (6.3)  and  (6.4). 

y  “  1/x  (6.3) 

and 

H  -  (II  x.)“1/ry“iexp{4'(h)}  ,  (6.4) 

i-1  1 

where  x^  is  observation  i,  n  is  the  number  of  observations,  exp(.) 

A 

represents  e  raised  to  the  power  ( . ) ,  and  ¥ (h)  is  given  by 

¥(h)  -  £n(h  -  0.5)  +  l/{24(h  -  0.5)2}  .  (6.5) 

^  ^  A 

The  approximation  in  Equation  (6.5)  is  only  valid  if  h  *  2.  Since  h 
appears  on  both  sides  of  Equation  (6.4),  a  few  iterations  using  trial 
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and  error  or  bisection  are  required  to  solve  for  it.  For  the  data  in 
Table  6.8,  the  estimates  are  u  “  0.552  aircraft  per  minute  or  y  *  33.15 

A 

aircraft  per  hour  and  h  »  19.6. 

Both  methods  yield  estimates  of  the  Erlang  number  that  are 
infeasible.  White,  Schmidt,  and  Bennett  recommend  choosing  an  integer 

A 

on  either  side  of  h  when  this  occurs.  The  estimate  for  the  data  in 

A  A 

Table  6.8  is  thus  either  h  ■  19  or  h  ■  20. 

The  final  step  in  fitting  the  data  in  Table  6.8  is  to  test  the 
hypothesis  that  the  sample  data  could  have  come  from  an  Erlang  distri¬ 
bution  with  the  estimated  parameters.  A  chi-square  goodness  of  fit 
test  will  be  used.  White,  Schmidt,  and  Bennett  suggest  that  maximum 
likelihood  estimates  be  used  for  the  parameters  of  the  Erlang  distribu¬ 
tion  to  gnnrartee  that  the  asymptotic  distribution  of  the  test 
statistic  is  chi-square.  The  test  is: 

H^:  The  data  come  from  an  Erlang  (y,h)  population. 

H^:  The  data  come  from  some  other  population. 

2  k  2 

Test  Statistic:  X  “  I  (0.  -  E.)  /E.  . 

i-1  1  1 

2  2 

Rejection  Region:  x  >  W) 

k  is  the  number  of  class  intervals  into  which  the  data  are  divided. 

0^  is  the  number  of  observations  that  fall  in  interval  i,  while  is 
the  expected  number  of  observations  that  would  fall  in  interval  i  under 
Hq.  a  is  the  level  of  significance  of  the  test.  The  number  of  degrees 
of  freedom  of  the  test,  d,  is  k  -  1  minus  the  number  of  estimated 
parameters  of  the  distribution.  Since  two  parameters  are  estimated 
for  the  Erlang  distribution,  d  is  k  -  3.  White,  Schmidt,  and  Bennett 
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recommend  that: 

1)  The  number  of  intervals  be  chosen  such  that  0^  —  5 
and  Ei  ■  5  for  all  i. 

2)  The  intervals  be  chosen  so  that  the  probabilities  of 

an  observation  falling  in  each  are  nearly  equal  under  H^. 

For  the  data  in  Table  6.8,  k  was  arbitrarily  chosen  to  be  six. 

Since  there  are  67  observations,  such  a  value  of  k  should  allow  the 

intervals  to  be  chosen  so  that  0^  and  are  at  least  five  for  each. 

A  computer  program  was  written  to  evaluate  the  integral  of  the  Erlang 

density  so  that  the  location  of  the  intervals  could  be  chosen  to 

satisfy  both  recommendations.  The  location  of  the  intervals  used  and 

the  values  of  0^  and  E^  when  the  hypothesized  distribution  is  an  Erlang 

(p-  33.15,  h  -  19)  distribution  are  given  in  Table  6.9.  If  a  level  of 

significance  a  ■  0.05  is  used,  the  critical  value,  X2  (3),  is  7.81. 

0.95 

Since  the  value  of  the  test  statistic  is  0.805  for  this  test,  the 
hypothesized  distribution  cannot  be  rejected.  In  addition,  the 
extremely  small  value  of  the  test  statistic  indicates  a  very  good  fit. 

TABLE  6.9 

Observed  and  Expected  Class  Frequencies  When  an  Erlang 
(U  *  33.15,  h  «  19)  Distribution  Is  Hypothesized 


Interval 


0.0 

1.385 

1.585 

1.735 

1.905 

2.205 

to 

to 

to ' 

to 

to 

to 

1.385 

1.585 

1.735 

1.905 

2.205 

oo 

10 

13 

9 

9 

14 

12 

10.02 

10.92 

9.72 

10.77 

14.36 

11.21 
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The  location  of  the  intervals  used  and  the  values  of  0^  and 
when  the  hypothesized  distribution  is  an  Erlang  (y  -  33.15,  h  -  20) 
distribution  are  given  in  Table  6.10.  The  value  of  the  test  statistic 
for  this  data  if  0.998.  Again,  if  the  level  of  significance  is  0.05, 
the  hypothesized  distribution  cannot  be  rejected  and  a  very  good  fit 
is  indicated. 


TABLE  6.10 

Observed  and  Expected  Class  Frequencies  When  an  Erlang 
(y  ■  33.15,  h  ■  20)  Distribution  Is  Hypothesized 


0.0 

to 

1.385 

1.385 

to 

1.585 

1.585 

to 

1.735 

Interval 

1.735 

to 

1.905 

1.905 

to 

2.205 

2.205 

to 

oo 

oi  10 

13 

9 

9 

14 

12 

E.  9.54 

11.00 

9.94 

11.04 

14.62 

10.86 

i 

Thus,  both  19  and  20  are  reasonable  estimates  of  the  Erlang 

number  for  the  data  in  Table  6.8.  Since  the  number  of  states  needed 

for  policy  iteration  is  n*h  +  1,  there  is  an  incentive  to  choose  the 

smaller  h.  Recall  that  each  step  of  the  policy  iteration  algorithm  or 

the  evaluation  of  each  policy  requires  the  inversion  of  a  matrix  that 

is  (n*h  +  1) -square.  In  light  of  this  requirement,  it  may  be  even  more 
s 

practical  to  use  an  estimate  of  h  that  Is  smaller  than  the  maximum 
likelihood  estimate  but  which  passes  the  goodness  of  fit  test.  While 
in  theory,  this  procedure  may  be  hard  to  justify,  it  should  not  intro¬ 
duce  any  appreciable  errors.  To  illustrate,  an  Erlang  (y  «  33.15, 
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2 

h  ■  8)  distribution  yields  x  “8.65  which  fails  the  chi-square  test  at 
ct  •  0.05,  but  passes  for  a  ■  0.01.  For  the  purpose  of  this  chapter, 
however,  the  Erlang  (y  ■  33.15,  h  ■  19)  distribution  is  chosen.  When 
Model  II  is  used  in  this  chapter,  an  exponential  distribution  with 
Vi  ■  33.15  is  assumed. 

Data  were  also  taken  at  the  Washington  National  Airport  on 
10  July  1979,  again  when  the  weather  was  good.  During  the  period  of 
observation,  Runway  18  was  being  used  for  both  arrivals  and  departures. 
Some  of  the  observations  of  service  time  were  made  using  a  radar  scope, 
but  the  majority  of  them  were  made  visually.  The  Washington  data  given 
in  Table  6.11  is  presented  primarily  to  support  the  general  applica¬ 
bility  of  the  Pittsburgh  data.  At  a  level  of  significance  of  0.05,  an 
Erlang  (y  ■  37.48,  h  ■  16)  distribution  provides  a  good  fit  of  the  34 
observations  taken. 

6.3  Deterministic  Service  Time  Model 

n  ,  the  Individual  optimum  balking  point  for  class  m  (in  terms 
s 

m 

of  customers),  is  found  using  Equation  (5.5).  The  value  of  n  is 

s 

m 

given  for  each  class,  m,  in  Table  6.12.  These  results  imply  that  for 
the  Erlang  19  model,  n*  is  18,  so  that  the  number  of  states  required 

Q 

is  343.  The  solution  techniques  based  on  policy  iteration,  Algorithm 
5.1,  will  require  the  inversion  of  a  matrix  that  is  343  rows  by  343 
columns.  Since  the  matrix  generated  will  be  sparse,  special  techniques 
could  be  used  to  Invert  such  a  large  matrix  while  maintaining  a 
reasonable  degree  of  numerical  precision.  However,  another  approach 
is  pursued  in  this  paper. 
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TABLE  6.11 

Observed  Service  Times  (Washington  National) 


Observation 

Time 

(min) 

Observation 

Time 

(Min) 

Observation 

Time 

(Min) 

1 

1.78 

1 

13 

2.12 

24 

2.37 

2 

1.40 

14 

1.60 

25 

2.15 

3 

1.63 

15 

1.12 

26 

1.77 

4 

1.60 

16 

1.52 

27 

1.22 

5 

1.95 

17 

1.12 

28 

2.18 

6 

1.70 

18 

1.48 

29 

2.32 

7 

1.66 

19 

2.05 

30 

1.62 

8 

1.15 

20 

0.95 

31 

1.16 

9 

1.30 

21 

1.97 

32 

1.82 

10 

1.15 

22 

1.57 

33 

1.29 

11 

1.30 

23 

1.00 

34 

1.95 

12 

1.48 
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TABLE  6.12 

Individual  Optimum  Balking  Points  (h  ■  19) 


< 

J 


Class 


n  (Customers) 
s 

m 


1 

2 

3 

4 

5 


18 

17 

14 

14 

12 


The  optimal  policy  and  corresponding  gain  rate  for  the  Erlang  19 
model  will  be  estimated  based  on  lower  Erlang  number  models.  The  next 
two  sections  develop  a  pattern  to  the  results  for  the  lower  Erlang 
number  models  which  suggests  that  this  estimation  procedure  is  reason¬ 
able.  Erlang  1  through  Erlang  8  models  will  be  used  for  this  process. 
Furthermore,  since  the  Erlang  «  service  time  distribution  represents 
deterministic  service  times,  a  deterministic  service  time  model  can  be 
used  as  one  bound  for  the  Erlang  19  results,  while  the  Erlang  8  results 
represent  the  other  bound  (the  Erlang  19  results  lie  between  these 
bounds).  Adler  and  Naor  (1969)  study  a  single  class  model  like  Model  I 
but  with  deterministic  service  times.  Their  results  are  presented  here 
and  then  used  in  Section  6.5  to  approximate  the  results  for  a  multiclass 
deterministic  service  time  model. 

Let  T  be  the  length  of  a  service.  F  is  the  steady  state 

probability  that  the  system  is  empty,  while  0  is  the  steady  state 

c 
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probability  that  the  service  station  Is  closed.  (0c  corresponds  to 

0  In  Model  I.)  The  average  rate  at  which  customers  depart  the 
n 

o 

service  station  Is  (1-0  )/T.  The  average  rate  at  which  customers 

o 

are  admitted  to  the  service  station  is  X (1  -  0  ).  In  steady  state, 

c 

#  these  two  rates  must  be  equal;  that  is, 

'  (1  -  0  )/T  -  \(1  -  0  )  .  (6.6) 

o  c 

The  social  optimum  objective  function  that  Adler  and  Naor  use  is 
I 

g  -  {R(l  -  0  )/T}  -  CL  .  (6.7) 

o 

From  Equation  (6.6),  (1  -  0q)/T  ■  X(1  0C)  “X1,  the  effective  arrival 

rate.  With  this  substitution,  Equation  (6.7)  becomes 

g  -  X'R  -  CL  (6.8) 

which  is  Equation  (3.2),  Ignoring  the  dependence  of  n.  Thus,  the  model 
of  Adler  and  Naor  is  Indeed  the  same  as  Model  I  except  for  the  deter¬ 
ministic  nature  of  the  service  times. 

Adler  and  Naor  define  a  state  space  that  can  assume  nonintegral 
values.  Recall  that,  in  Model  I,  the  state  of  the  system  is  the  number 
of  customers  present  in  the  system.  Adler  and  Naor  define  the  state  of 
the  system  to  be  the  number  of  whole  service  times,  T,  present  in  the 

*  system.  (They  assume  an  arrival  can  determine  how  much  service  time 

*  remains  for  the  customer  in  service.)  Since  an  arrival  can  find  the 

service  of  the  customer  in  service  partially  completed,  the  state  of 

the  system  can  take  on  nonintegral  values. 

The  balking  point  for  the  individual  optimum  problem  is  v  -  1, 

s 

where  v  is  determined  by 
8 
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v  -  R/(CT)  .  (6.9) 

s 

The  authors  show  that  the  forced  balking  point  for  the  social  optimum 
problem  is  given  by  vq  -  1,  where  vQ  satisfies 

va0o(vo.P)  “  %  +  L<v0»  "  0  *  (6.10) 

dler  and  Naor  also  show  that 

0q(v»P)  ■  l/(l  +  I  (-l)^”1  -^i)l —  P  exp{  (v  -  j)p}) 

(6.11) 

and 

L(v,p)  -  n  -  0Q(v,p)j  exp{ (v  -  l)p)(l  +  np  -  vp) 

+  ?  C«xp{(v  -  J)p> 

j-2  '‘k-l  L) 

+  (-i)J-l  iiv(~  (1  +  np  -  vp)})} 

(6.12) 

where  n  Is  the  greatest  integer  in  v.  One  way  to  solve  for  vq  Is  to 

guess  a  starting  value  such  as  v  ■  v  ,  then  use  a  technique  like 

s 

bisection  to  search  for  a  value  of  v  that  satisfies  Equation  (6.10), 
where  Equations  (6.11)  and  (6.12)  are  used  to  evaluate  0^  and  L 
required  In  Equation  (6.10).  Once  a  value  of  vq  is  determined,  the 
corresponding  values  of  0^  and  L  can  be  used  in  Equation  (6.7)  to 
determine  the  maximum  gain  rate  of  the  model. 

To  illustrate  the  above  solution  technique,  let  R  •  5,  C  -  2, 

X  ■  1,  and  T  ■  1/3  be  the  parameters  of  the  model.  The  traffic 
intensity,  p,  is  given  by  p  ■  XT  ■  1/3.  Equation  (6.9)  yields  v  -  7.5. 

S 

After  a  few  iterations,  bisection  yields  vq  -  5.4167,  which  implies  that 
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the  forced  balking  point  is  A. 4167.  Equations  (6.11)  and  (6.12)  yield 
0q  ■  0.6667  and  L  ■  0.4164.  Finally,  Equation  (6.7)  gives  g  ■  4.1667. 

To  illustrate  the  suggested  method  for  bounding  the  results  for 
an  Erlang  19  model,  the  deterministic  results  will  be  compared  with 
the  results  of  several  Erlang  models  with  low  values  of  h.  Since 
Adler  and  Naor  assume  that  an  arrival  knows  how  much  service  time 
remains  for  the  customer  in  service,  the  most  appropriate  Erlang  model 
for  comparison  with  the  deterministic  model  is  one  in  which  the 
customer  knows  the  service  phase  of  the  customer  in  service.  Table 
6.13  gives  the  social  optimum  gain  rate  and  forced  balking  points  for 
several  Erlang  models  of  the  example.  The  forced  balking  point  in 
Table  6.13  is  given  in  terms  of  customers  in  line  plus  phases  of  service 
left  for  the  customer  in  service.  For  instance,  4  cust  +  3  ph  means 
that  customers  are  forced  to  balk  if  the  state  of  the  system  represents 
at  least  four  customers  in  line  plus  three  phases  of  service  remaining 
for  the  customer  in  service.  The  deterministic  results  plus  the  data 
in  Table  6.13  indicate  that  the  optimal  policy  for  an  Erlang  19  model 
is  probably  a  balking  point  of  four  customers  in  line  plus  several 
phases  of  service  left  for  the  customer  in  service  which  should  yield 
a  gain  rate  between  4.146  and  4.1667.  The  actual  phase  optimum 
solution  of  the  Erlang  19  model  provides  a  gain  rate  oi  4.158  with  a 
balking  point  of  four  customers  in  line  plus  eight  phases  of  service 
left  for  the  customer  in  service. 
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TABLE  6.13 


Results  for  Several  Erlang  Models 
of  the  Example 


Erlang  Number 

Gain  Rate 

Forced  Balking  Point 

1 

4.003 

4  cust  +  1  ph 

2 

4.084 

4  cust  +  1  ph 

6 

4.139 

4  cust  +  3  ph 

8 

4.146 

4  cust  +  4  ph 

6.4  Results  for  Model  II 

For  Model  II,  applied  to  the  Pittsburgh  Airport  problem,  the 

service  rate  of  the  exponential  server  is  33.15  aircraft  per  hour.  The 

computer  program  in  Appendix  D,  POLIT,  was  used  to  solve  for  the  optimal 

policy,  the  resulting  gain  rate,  and  the  probability  of  rejection  oi  an 

arrival  from  each  class.  The  probability  of  rejection  of  class  m  is 

found  by  summing  the  steady  state  probabilities  of  the  system  occupying 

states  n  through  n*.  The  results  from  POLIT  are  summarized  in 
o  s 

m 

Table  6.14. 

Since  the  first  iteration  of  the  policy  iteration  algorithm 
chooses  the  individual  optimum  policy  (see  Table  6.12),  its  gain  rate 
can  be  found  ($4905  per  hour).  Thus,  the  model  indicates  that  for  a 
peak  traffic  hour,  the  average  net  benefit  to  the  airline  companies 
can  be  increased  $1784  by  implementing  the  social  optimum  policy 
rather  than  the  individual  optimum  policy. 


TABLE  6.14 


Results  for  the  Exponential  (y  ■  33.15)  Model 


Rejection  Probability 


<  0.01 


<  0.01 


<  0.01 


<  0.01 


0.316 


Gain  Rate  ($/Hr) 


Although  the  model  assigns  no  cost  to  rejecting  an  aircraft,  the 
probability  of  rejection  of  a  clar  indicates  how  the  optimal  policy 
would  affect  the  current  operatic  ,  of  that  class  at  the  airport.  It 
is  possible  to  use  the  model  so  that  a  "cost"  of  rejection  is  introduced 
by  setting  controls  on  the  maximum  probability  of  rejection  allowed  for 
any  class.  This  approach  might  consist  of  a  procedure  to  search  various 
combinations  of  class  arrival  rates  using  POLIT  to  evaluate  each  combin¬ 
ation  until  a  combination  is  achieved  that  produces  both  an  acceptable 
gain  rate  and  an  acceptable  maximum  probability  of  rejection.  The 
schedule  used  in  the  Pittsburgh  example  leads  to  a  rejection  probability 
of  0.316  for  the  class  that  provides  65%  of  the  arrivals  at  the  airport. 
The  procedure  mentioned  above  could  be  used  to  improve  on  this  situation 
Although  such  use  of  the  model  is  beyond  the  scope  of  this  chapter,  the 
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sensitivity  of  the  gain  rate  and  rejection  probabilities  to  deviations 
from  the  optimal  policy  will  be  checked  to  provide  some  insight  into 
how  this  use  of  the  model  might  proceed. 

A  straightforward  approach  to  checking  the  sensitivity  of  the 
model  to  deviations  from  the  optimal  policy  is  to  use  the  policy 
evaluation  portion  of  the  policy  iteration  algorithm  to  evaluate 
several  policies  that  are  in  some  sense  close  to  the  optimum.  The 
results  for  several  such  policies  are  given  in  Table  6.15.  For  con¬ 
venience,  a  balking  point  of  14  rather  than  13  is  used  for  class  one. 
Since  the  steady  state  probability  of  the  system  occupying  state  14  is 

essentially  zero,  such  a  change  from  n  *13  has  little  effect  on  the 

°1 

results  of  the  model.  Results  like  those  presented  in  Table  6.14  pro¬ 
vide  an  administrator  some  capability  to  trade  off  a  reduction  in  the 
gain  rate  for  a  reduction  in  the  maximum  probability  of  rejection  for 


any  class. 

The  sensitivity  of  the  results  to  changes  in  a  single  parameter 
was  tested  by  determining  the  range  of  parameter  values  for  which  the 
optimal  policy  remained  the  same.  The  service  rate,  y,  was  varied  both 
up  and  down  until  the  optimal  policy  changed.  The  results  are  given  in 


Table  6.16.  Since  the  change  of  n 

°2 


to  the  low  probability  (less  than  10 


from  14  to  13  is  insignificant  due 
■4 

)  of  the  system  occupying  state 


13  or  14,  the  optimal  policy  is  stable  for  y  in  the  range  31+  to  34-. 


The  location  of  the  change  could  be  found  more  precisely  with  additional 


evaluations. 


V  P3 
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TABLE  6.15 

Results  for  the  Exponential  (y  ■  33.15) 
Model  for  Several  Nonop timum  Policies 


Class 

Balking 

Point 

- r 

Rejection 
Probability  j 

Balking 

Point 

Rejection 

Probability 

1 

14 

<  0.01 

14 

<  0.01 

2 

14 

1 

<  0.01 

14 

<  0.01 

3 

! 

6 

0.021 

6 

0.052 

4 

7 

<  0.01 

7 

<  0.01 

5 

4 

0.250 

5 

0.197 

Gain 

r) 


6674 


6564 


Class 

Balking 

Point 

....  1 

Rejection  1 

Probability 

Balking 

Point 

Rejection 

Probability 

1 

14 

<  0.01 

1 

14 

<  0.01 

2 

14 

<  0.01 

14 

<  0.01 

3 

6 

0  131 

7 

0.112 

4 

6 

0  *31 

7 

0.112 

5 

6 

0.131 

7 

0.112 

Gain 
Rate 
($/Kr) 


6396 


6214 
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TABLE  6.16 

Balking  Points  and  Gain  Rate 
for  Various  Values  of  y 


Class 

31 

32 

33.15 

34 

1 

13 

13 

13 

14 

2 

13 

13 

14 

14 

3  ' 

5 

i 

6 

6 

6 

4 

!  7 

7 

7 

8 

5 

!  3 

3 

3 

4 

Gain 

Rate 

($/Hr) 

6367 

6521 

6689 

6811 

The  value  of  A^,  the  arrival  rate  of  class  five,  was  also  varied 
in  both  directions  until  the  optimal  policy  changed.  The  results  are 
given  in  Table  6.17.  These  results  indicate  that  the  optimal  policy  is 
stable  for  A^  in  the  range  19+  to  27-. 

Finally,  the  value  of  R,.,  the  reward  for  the  service  of  a  class 
five  customer,  was  varied  until  the  optimal  policy  changed.  The  results 
given  in  Table  6.18  indicate  that  the  optimal  policy  is  stable  for  R5 

4*  «» 

in  the  range  230  to  270  . 

If  desired,  the  stability  of  the  optimal  policy  to  changes  in 

the  other  parameters,  including  the  C  '  s,  can  also  be  tested.  The 

m 

technique  used  for  y,  A^,  and  R^  can  be  applied  to  these  determinations 


as  well. 


TABLE  6.17 

Balking  Points  and  Gain  Rate 
for  Various  Values  of  X. 


5 

20.90 


in 

te 

fHr) 

6568 

663: 

L  6689 

6958 

TABLE  6.18 


Balking  Points  and  Gain  Rate 
for  Various  Values  of  R^ 


Rate  ($/Hr) 


6.5 


Results  for  Model  III 


For  Model  III,  the  service  time  distribution  Is  modeled  as  an 
Erlang  (y  »  33.15,  h  -  19)  distribution  as  developed  in  Section  6.2. 

As  this  distribution  leads  to  a  large  number  of  states  (343),  the 
solution  presented  in  this  section  is  found  in  a  somewhat  roundabout 
manner.  First,  Erlang  models  with  U  ■  33.15  and  h  ■  2,  4,  6,  and  8 
are  solved  using  Algorithm  5.1.  These  results  are  used  to  hypothesize 
the  optimal  policy  for  the  Erlang  19  model.  The  gain  rate  of  the 
Erlang  8  model  serves  as  a  lower  bound  for  that  of  the  Erlang  19  model. 
Then,  the  deterministic  model  of  Section  6.3  is  used  to  approximate  the 
gain  rate  for  a  deterministic  service  time  model  of  the  Pittsburgh 
Airport.  This  gain  rate  serves  as  an  upper  bound  on  the  gain  rate  for 
the  Erlang  19  model. 

The  results  for  the  lower  Erlang  number  models  will  be  presented 
first.  Recall  that  the  policy  iteration-based  algorithm,  Algorithm  5.1, 
uses  as  a  starting  point  the  results  from  an  Erlang  model,  the  phase 
optimum  model,  in  which  customers  can  determine  the  service  phase  of 
the  customer  in  service.  The  phase  optimum  results  for  Erlang  models 
with  y  ■  33.15  and  h  ■  2,  4,  6,  and  8  are  given  in  Table  6.19.  As  in 
Table  6.13,  the  forced  balking  points  are  given  in  terms  of  customers 
in  line  plus  phases  of  service  left  for  the  customer  in  service. 

Recall  that  an  implementable  or  linked  policy  does  not  require 
the  administrator  to  determine  the  service  phase  of  the  customer  in 
service.  The  set  of  candidate  forced  balking  points,  found  using 
Step  (c)  of  Algorithm  5.1,  are  given  in  Table  6.20  for  each  of  the 
four  linked  Erlang  models  considered.  Except  for  class  three  in  the 


TABLE  6.19 


Phase  Optimum  Balking  Points  and  Gain  Rate 
for  Several  Low  Erlang  Number  Models 


TABLE  6.20 

Candidate  Forced  Balking  Points  for 
Several  Linked  Erlang  Models 


2 

4 

13,14 

13,14 

14 

14 

6 

5,6 

7,8 

7,8 

3,4 

3,4 

h 


6 

8 

13,14 

13,14 

14 

14 

5,6 

5,6 

7,8 

7,8 

3,4 

3,4 

Erlang  two  model,  the  candidate  forced  balking  points  for  each  class 
are  the  same  for  all  four  Erlang  models.  Since  the  steady  state 
probability  of  any  of  those  systems  occupying  13  or  14  la  insignifi¬ 
cant,  the  forced  balking  point  for  class  one  is  arbitrarily  chosen  to 
be  14.  Policies  that  use  all  possible  combinations  of  this  reduced 
set  of  candidate  balking  points  were  evaluated  to  determine  the  best 
linked  policy  for  each  model.  The  results  for  the  best  policy  for 
each  model  are  given  in  Table  6.21.  For  each  of  these  models,  several 
policies  tied  (within  roundoff  error)  for  the  best  policy.  Of  these, 

the  policy  with  the  largest  values  of  the  n  was  chosen  in  each  case 

o 

m 

as  the  best  policy.  The  ties  were  the  result  of  low  probabilities  of 
the  systems  occupying  states  beyond  state  six.  Note  that  for  each 
model,  the  best  linked  policy  found  using  Algorithm  5.1  achieves  a  gain 
rate  that  is  better  than  99.8%  of  the  phase  optimum  gain  rate.  Depend¬ 
ing  on  the  use  of  the  model,  such  a  result  probably  precludes  the  need 
for  a  more  lengthy  search  for  a  best  policy.  Since  the  same  linked 
policy  provides  the  best  results  for  all  four  models,  it  seems  reason¬ 
able  to  assume  that  the  policy  will  also  provide  the  best  results  for 
the  Erlang  19  model.  Although  the  gain  rate  increases  with  the  Erlang 
number  h,  the  incremental  jump  decreases  with  higher  values  of  h. 

Thus,  the  gain  rate  of  the  Erlang  eight  model,  $7223  per  hour,  provides 
a  good  lower  bound  on  the  gain  rate  of  the  Erlang  19  model. 

A  deterministic  service  time  model  provides  the  capability  to 
get  an  upper  bound  on  the  gain  rate  for  the  Erlang  19  model;  however, 
the  deterministic  model  of  Adler  and  Naor  (1969)  presented  in  Section 
6.3  only  applies  to  a  single  class  of  customers.  No  attempt  is  made 
here  to  extend  their  work.  Instead,  the  single  class  deterministic 


TABLE  6.21 


Best  Results  for  Several  Linked  Erlang  Models 


h_ 

-  2 

h  ■ 

_4 

Class 

Balking 

Point 

Rejection 

Probability 

Balking 

Point 

Rejection 

Probability 

1 

14 

<  0.01 

. 

14 

<  0.01 

2 

14 

<  0.01 

14 

<  0.01 

3 

6 

<  0.01 

6 

<  0.01 

4 

8 

<  0.01 

8 

<  0.01 

5 

3 

0.2796 

3 

0.2557 

Gain 

Rate  ($/Hr) 


6963 


7129 


Class 

h  « 

Balking 

Point 

_6 

Rejection 

Probability 

Balking 

Point 

_8 

Rejection 

Probability 

1 

14 

<  0.01 

14 

<  0.01 

2 

14 

<  0.01 

14 

<0.01 

3 

6 

<  0.01 

6 

<  0.01 

4 

8 

<  0.01 

8 

<  0.01 

5 

3 

0.2464 

3 

0.2414 

Gain 

Rate  ($/Hr) 


7191 


7223 
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model  Is  used  to  approximate  the  gain  rate  £or  the  five  class 
Pittsburgh  Airport  model  with  deterministic  service  times.  The  five 
classes  of  aircraft  are  replaced  by  a  single  composite  class  whose 
reward  for  a  service  is  determined  by  a  weighted  average  of  the  rewards 
for  the  five  classes.  The  weight  for  each  class  can  be  determined  from 
Table  6.2  as  the  proportion  of  the  total  arrival  rate  that  is  accounted 
for  by  the  class.  Analogously,  the  cost  per  hour  of  flight  for  the 
composite  class  is  a  weighted  average  of  the  costs  for  each  of  the 
five  classes.  The  arrival  rate  for  the  composite  class  is  the  sum  of 
the  arrival  rates  of  all  five  classes.  Since  the  services  are 
deterministic,  the  service  rate  is  a  constant  whose  value  is  the 
estimate  of  y  used  in  Models  II  and  III.  The  parameters  of  the 
composite  class  are  given  in  Table  6.22. 


TABLE  6.22 

Parameters  for  the  Single  Composite  Class  Model 


Parameter 


Value 


R  306,7 

C  '  767.9 

X  32.0 


U 


33.15 
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The  approach  used  here  to  estimate  the  gain  rate  for  a  determin¬ 
istic  five  class  model  from  the  deterministic  single  composite  class 
model  begins  with  the  comparison  of  the  gain  rates  found  by  policy 
iteration  for  several  phase  optimum  Erlang  models  applied  first  to  the 
composite  class  and  then  to  the  full  five  classes.  Calculation  of  the 
ratio  of  the  composite  class  gain  rate  to  the  ?ive  class  gain  rate  for 
Models  II  and  III  with  h  ■  2,  4,  6,  and  8  provides  a  means  of  estimating 
the  ratio  of  the  composite  class  deterministic  gain  rate  to  the  five 
class  deterministic  gain  rate.  Table  6.23  gives  the  composite  class 
gain  rate,  the  five  class  gain  rate,  and  the  ratio  of  the  two  rates  for 
each  of  the  five  models  mentioned. 


TABLE  6.23 

Policy  Iteration  Phase  Optimum  Results  for 
Several  Models  of  Both  a  Single  Composite 
Class  and  the  Full  Five  Classes 


Model 

• 

; 

Composite  Class 
*  Gain  Rate  ($/Hr) 

Five  Classes 

Gain  Rate  ($/Hr) 

Ratio 

(Composite/Five) 

II 

! 

i  6506 

6689 

0.973 

III,  h  - 

2 

6800 

6975 

0.975 

h  « 

4 

6977 

7141 

0.977 

h  - 

6 

7042 

7203 

0.978 

h  - 

8 

!  7076 

1 

7235 

0.978 
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The  deterministic  gain  rate  for  the  single  composite  class  can 
be  determined  using  the  results  of  Adler  and  Naor  presented  in  Section 
6.3.  Their  method  gives  0q  ■  0.1504  and  L  -  1.890  which,  when  substi¬ 
tuted  into  Equation  (6.7)  yields  g  ■  $7186  per  hour.  Since  Table  6.23 
suggests  that  0.97  is  a  good  lower  bound  on  the  ratio  of  the  determin¬ 
istic  composite  class  gain  rate  to  the  deterministic  five  class  gain 
rate,  a  reasonable  upper  bound  on  the  deterministic  five  class  gain 
rate  is  $7408  per  hour.  This  gain  rate  serves  as  an  upper  bound  on 
the  gain  rate  of  the  Erlang  19  model. 

The  results  of  this  section  suggest  that  the  socially  optimal 
policy  for  the  Erlang  19  model  of  the  Pittsburgh  Airport  is  to  implement 
the  following  set  of  forced  balking  points:  nQ  ■  (14,  14,  6,  8,  3). 

This  policy  should  provide  a  gain  rate  of  somewhere  between  $7223  and 
$7408  per  hour  to  the  arrivals  during  a  busy  period,  .while  incurring  a 
rejection  probability  of  less  than  0.2414  for  class  five  customers  and 
less  than  0.01  for  any  other  class.  Analysis  of  the  sensitivity  of 
these  results  to  changes  in  the  input  parameters  could  be  tested  in 
the  same  manner  as  was  done  for  Model  II. 

6.6  Conclusion 

This  chapter  was  devoted  to  applying  the  techniques  developed 
to  determine  the  optimum  control  policy  for  the  landing  queue  at  the 
Pittsburgh  Airport.  Use  was  made  of  Models  II  and  III.  The  parameters 
of  the  models  were  determined  from  published  FAA  data,  the  literature, 
and  direct  observation.  Computational  simplifications  were  investi¬ 
gated  and  sensitivity  of  solutions  examined.  All  indications  are  that 
the  simplified  analyses  are  practical  and  sensitivities  easy  to 


determine . 
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A  few  suggestions  for  refining  the  estimates  of  the  model 
parameters  follow;  however,  before  such  refinements  are  incorporated 
into  the  model,  it  is  suggested  that  the  sensitivity  of  the  model  to 
the  changes  be  checked.  It  might  be  possible  to  estimate  the  effect 
of  the  changes  based  on  model  sensitivities  already  determined. 

The  values  can  be  refined  by  using  an  average  passenger  load 

factor  for  each  type  of  aircraft  and  thus  for  each  class.  Further 

improvements  in  the  estimates  of  the  Rffl's  also  affect  the  estimates  of 

the  C  ' s  and  X  's  through  expansion  of  the  number  of  classes.  (Each 
m  m 

new  class  m  would  require  its  own  estimated  R  ,  C  ,  and  X  .)  For 

mm  m 

Instance,  the  existing  classes  can  be  subdivided  based  on  a  range  of 
values  of  the  passenger  load  factor.  Thus,  class  one  might  become 
four  classes,  one  for  a  0  -  25%  loading  factor,  another  for  a  26  -  50% 
loading,  and  so  on.  R^  might  be  improved  even  further  by  again 
subdividing  the  classes  based  on  individual  aircraft  type. 

The  estimates  of  the  X^  can  be  improved  by  basing  them  on 
actual  or  proposed  schedules  of  arrivals  during  representative  busy 
periods  rather  than  yearly  statistics. 

The  estimates  of  the  parameters  of  the  service  time  distribution 
were  based  on  observations  made  at  the  Pittsburgh  Airport.  The 
estimates  were  confirmed  by  a  second  set  of  observations  made  at  the 
Washington  National  Airport.  Several  additional  days  of  observations 
from  the  Pittsburgh  Airport  may  result  in  better  estimates  of  the 
parameters  of  the  service  time  distribution. 

Several  areas  for  further  study  were  broached  in  this  chapter. 
One  of  these  is  the  difficulty  in  solving  problems  involving  a  large 
number  of  states.  Special  techniques  for  solving  large  sparse  systems 
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of  linear  equations  could  be  examined  to  permit  direct  solution  of 
models  like  the  Erlang  19  model.  Another  problem  is  the  extension  of 
the  work  of  Adler  and  Naor  (1969)  to  several  classes  of  customers. 
While  this  might  require  substantial  analytical  work,  it  would  provide 
a  means  of  quickly  bounding  the  results  for  any  Erlang  model.  The 
final  problem  Introduced  Is  perhaps  the  most  Interesting.  The  problem 
Is  to  design  a  schedule  of  arrivals  that  achieves  an  acceptable  social 
gain  rate  while  keeping  the  maximum  rejection  probability  for  any 
class  of  customers  below  some  upper  bound.  Models  II  and  III  can  be 
used  to  solve  this  problem  as  a  part  of  a  search  technique  or  a  new 
expanded  model  can  be  developed. 


\ 


CHAPTER  VII 


OTHER  MODELS 

In  this  chapter,  three  new  models  are  developed  to  illustrate 
that  the  semi-Markov  approach  used  for  Models  I,  II,  and  III  can  be 
adapted  to  solve  other  related  models.  Although  all  three  models  have 
M  classes  of  customers  and  an  exponential  server,  the  models  could 
easily  be  generalized  to  an  Erlang  server.  The  three  models  considered 
are: 

1)  a  nonpreemptive  priority  service  discipline  model, 

2)  a  claaa  dependent  service  rate  model,  and 

3)  a  nonpreemptive  service  discipline  model  with  class 
dependent  service  rates. 

7 . 1  Nonpreemptive  Priority  Service  Discipline  Model 

The  assumptions  and  structure  of  this  model  are  those  of  Model  II 
except  that  the  service  discipline  utilizes  nonpreemptive  priorities 
rather  than  position  in  line  (first  come  first  served)  to  determine 
the  order  of  service.  Let  class  one  have  highest  priority,  class  two 
next  highest,  and  so  on.  Within  a  class,  the  customers  are  served  first 
come  first  served.  Since  the  priorities  are  nonpreemptive,  the  custom¬ 
er  Ln  service  is  allowed  to  finish  regardless  of  the  priority  of  an 
arrival . 

The  state  space  for  this  model  is  a  vector  a,  where  is  the 

class  of  the  customer  in  service  and  a  is  the  number  of  class  m 

m 

customers  in  the  system.  To  illustrate  the  state  space  consider  a 
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two  class  example  in  state  (2,1,2).  Thus,  there  are  two  customers  of 
class  two  and  one  of  class  one  in  the  system  with  one  of  the  class  two 
customers  in  service.  If  the  next  event  is  the  completion  of  a  service, 
the  3tate  becomes  (1,1,1)  since  the  class  one  customer  has  priority 
over  the  remaining  class  two  customer. 

Self-optimizing  customers  of  class  m  will  join  the  system  if 
their  expected  net  benefit  for  joining  is  greater  than  zero.  When  the 
state  of  the  system  is  a,  the  expected  net  benefit  of  joining  for  a 
customer  of  class  m  is 


R  -  t  (a)  C 
m  m  m 


(7.1) 


where  t  (a)  is  tne  time  the  customer  expects  to  spend  in  the  system 

HI 

when  he  finds  the  system  in  state  a.  A  class  one  arrival  must  wait  for 
all  class  one  customers  in  the  system  to  be  served  before  he  can  be 
served.  In  addition,  he  must  wait  for  the  customer  in  service  to 
finish;  however,  if  the  customer  in  service  is  a  member  of  class  one, 
he  has  already  been  counted  with  the  class  one  customers.  Eq  ation  (7.2) 
gives  t^(a) . 

M 

t  (a)  »  (a  +  1  +  l  6  )/li  ,  (7.2) 

1  1  m-2  m,a0 

where 


1, 

< 

lf  *0  ■  “ 

otherwise 

A  class  two  arrival  must  wait  for  all  class  one  and  class  two  customers 
in  the  system  to  be  served  before  he  can  be  served.  If  the  customer  in 
service  is  not  a  member  of  class  one  or  two,  one  more  customer  must  be 
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added  to  thy  list  of  customers  who  must  be  served  before  the  class  two 
arrival.  Thus, 


t2(a)  -  (a  +  a  +  1  +  I  «  )/U 

*  1  1  w-b  ffl’a0 


(7.3) 


In  general,  a  class  m  customer  must  wait  for  the  service  of  customers 
in  the  system  of  his  c':ass  and  all  classes  of  higher  priority.  In 
addition,  if  the  customer  in  service  is  of  lower  priority,  he  has  not 
been  counted  and  must  also  be  served  before  the  class  m  arrival.  Thus, 


"  <1  +  1  +  1  61  a  5 

i-1  1  j-m+1  0 


(7.4) 


Self -optimizing  customers  of  class  m  will  join  the  system  if 


m  M 

1  -  t  (a)C  «  R  -  (T  a.  +  1  +  T  6.  )C  /y  - 
“  "  “  m  i-l  1  Wi  ” 


If  n  is  such  that 
8m 


R-nC/y-0>R-(n  +  1)C  /y 


m  s  m 
m 


ms  m 

m 


then. 


[V/CJ 


(7.5) 


(7.6) 


where  the  brackets  indicate  the  greatest  integer  function,  n  is  the 

s 

m 

individual  optimum  balking  point  for  class  m  customers.  If  an  arrival 
from  class  m  finds 


l  *±  +  l  6i  a  <  ns 
i**l  j»nri-l  ^  ’  0  m 


(7.7) 


he  joins;  otherwise,  he  balks. 


;  1 


r  ’.v  .  .-P-V 
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Since  customers  are  served  on  the  basis  or  their  priority ,  the 
expected  net  reward  for  a  customer  of  class  m  who  is  in  the  system  but 
not  in  service  is  altered  by  later  arriving  customers  of  higher  priority 
who  join  the  system.  This  may  result  in  customers  desiring  to  renege. 
For  instance,  if  a  class  m  customer  joined  the  system  when 


m  M 

l  +  I  a 

i-1  1  j-mfl  J,C 


and  the  next  event  were  the  arrival  of  a  higher  priority  customer,  the 
class  m  customer  would  want  to  leave  the  system  to  cut  his  expected 
loss.  Even  though  his  time  in  the  system  has  cost  him  money,  he  now 
expects  to  lose  more  money  by  remaining  in  the  system.  If  customers 
are  allowed  to  renege,  another  problem  arises.  The  decision  to  renege 
by  a  class  m  customer  affects  the  reneging  decision  of  a  lower  priority 
customer.  Thus,  the  presence  or  absence  of  information  concerning  the 
reneging  decisions  of  customers  of  higher  priority  can  lead  a  customer 
to  make  different  decisions  concerning  reneging.  Several  individual 
optimum  models  would  result  from  different  combinations  of  assumptions 
about 

1)  whether  or  not  to  allow  reneging,  and 

2)  if  reneging  is  allowed,  whether  or  not  to  assume  that 
each  customer  knows  the  decision  of  every  other 
customer  when  he  makes  his  own  decision  about  renege  -,g. 

Since  the  social  optimum  problem  is  of  principal  interest,  th^se  ideas 
will  not  be  pursued  here. 

As  will  be  discussed  later,  the  individual  optimum  balking  point 
for  each  class  serves  as  an  upper  bound  on  the  social  optimum  balking 
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point  for  the  class.  This  again  provides  a  bound  on  the  size  of  the 

state  space  required  for  the  semi-Markov  decision  process  formulation 

of  the  social  optimum  problem.  Since  self-optimizing  customers  of 

class  m  can  join  if  as  many  as  n  -  1  customers  of  their  own  class 

am 

are  in  the  system  (this  can  occur  if  no  customers  of  higher  priority 

are  in  the  system  and  a  class  m  customer  is  in  service) ,  a  bound  on 

the  size  of  the  state  space  for  the  social  optimum  problem  is  the 

number  of  states  such  that  a  *  n  ,  for  all  m.  Let  a*  be  the  set  of 

m  s 

m 

all  such  states. 

Recall  that  Section  5.4  points  out  that  the  social  optimum 
problem  of  Model  III  (as  well  as  Models  I  and  II)  differs  from  the 
self -optimum  problem  in  that  a  self-optimizing  customer  fails  to 
consider  the  decrease  in  expected  benefits  to  later  arriving  customers 
caused  by  his  joining  the  queue.  This  external  economic  effect  also 
occurs  in  this  model  but  manifests  itself  in  two  ways.  Not  only  does 
the  self-optimizing  customer  fail  to  consider  the  decrease  in  benefits 
to  later  arriving  customers  of  lower  priority  that  his  joining  causes, 
but  he  also  fails  to  consider  the  decrease  in  benefits  to  customers  of 
lower  priority  who  are  already  in  the  system  (but  not  in  service) .  The 
formulation  of  the  social  optimum  problem  considers  both  forms  of  the 
external  economic  effect. 

Consider  an  administrator  who  charges  a  joining  customer  for  the 
expected  costs  his  joining  causes  customers  present  in  the  system  as 

^Since  the  optimal  policy  has  not  been  shown  to  be  a  control- 
limit  policy,  nQ  needs  to  be  redefined  a*?  one  plus  the  maximum  number 
of  customers  in  °  the  system  for  which  a  class  m  arrival  joins  the 
queue . 
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well  as  for  his  own  expected  costs.  Since  the  joining  of  a  lowest 
priority  (class  M)  arrival  affects  no  customers  already  in  the  system, 
the  expected  system  gain  in  state  a  due  to  his  joining  is 


«h  -  +  “V 


(7.8) 


The  joining  of  a  customer  from  class  M-l  causes  all  customers  from 
class  M  who  are  in  the  queue  to  remain  in  the  system  for  an  additional 
service.  Thus,  the  expected  system  gain  in  state  a  due  to  a  class  M-l 
customer  joining  is 

^  -r  "  ai  +  1  +  ^M.a^Si-l^  (aM  “  5M,a0)CM/y 

(7.9) 

In  general,  the  expected  system  gain  in  state  a  due  to  a  class  m 
customer  joining  is 

IQ  M  M 

\  -  <1  \  +  1  +  I  «.  a  -  I  (a  -  6  JC  /Vl  ■ 

“  i-1  1  j-m+1  J,a0  m  j-mfl  J  J,a0  J 


(7.10) 


This  approach  offers  a  means  of  internalizing  the  cost  to  customers 
already  in  the  system  and  avoids  reneging  since  a  customer's  expected 
costs  are  known  at  the  time  of  his  arrival  and  do  not  change  with 
later  arriving  customers. 

The  other  portion  of  the  external  economic  effect,  the  decrease 
in  benefits  to  later  arriving  customers  of  lower  priority  caused  by  a 
customer  joining  the  system,  is  internalized  as  in  Models  I,  II  and  III 
through  the  social  optimum  objective  function  which  is 


PeC  AC. 
s  a 


l  £ 


(7.11) 


Pec  Ac,  -  .  a  a 

s  d  aea* 
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Again,  only  stationary  Markovian  decisions  are  considered  and  the  work 

of  Der  "  '1962)  is  used  to  limit  the  search  to  a  nonrandomized  rule. 

P  “ 

0_  is  the  steady  state  probability  of  the  system  occupying  state  a  under 

a  P 

policy  P.  q__  is  the  expected  gain  per  unit  time  the  system  is  in  state 
a 

a  when  policy  P  is  in  effect.  The  inclusion  of  the  steady  state 
probabilities  of  occupying  the  various  states  allows  the  administrator 
to  consider  the  expected  decrease  in  benefits  to  later  arriving 
customers  of  lower  priority  when  deciding  whether  or  not  to  admit  a 
customer.  Equation  (7.11)  can  be  rewritten  as 


max 

PeC  A C . 
s  d 


8P 


max 

PeC  A C, 
s  d 


(7.12) 


where  k  (a)  is  an  indicator  that  is  one  if  a  class  m  arrival  is  admitted 
m 

to  the  system  when  the  system  is  in  state  a  and  zero  if  not.  f^a) , 
which  is  the  expected  system  gain  in  state  a  if  a  class  m  customer  is 
admitted,  is  given  by  Equation  (7.10), 


f  (a) 
in 


-  <1  +  1  +  l 

m  i-1  1 


l  a  >VV 

j-m+l  J,a0 


M 

l  (aj 

j-m+l  J 


)Cj/y 


(7.13) 


The  formulation  expressed  in  Equation  (7.12)  can  be  solved  by  policy 

iteration  as  will  be  demonstrated  by  example.  Theorem  4.3  can  be 

adapted  to  show  that  n  «  n  •  No  attempt  is  made  here  to  establish 

o  s 
m  m 

the  form  of  the  optimal  policy. 

Consider  a  two  class  example  with  y  -  4  and  \^  -  2,  «  2,  and 

C  -  3.  Also,  X2  ■  4,  R.^  -  1,  and  *  2.5.  From  Equation  (7.6), 

n  -  2  and  n  -  1.  Thus, 

81  s2 


a*  -{(0,0,0),  (1,1,0),  (2,0,1),  (1,2,0),  (1,1,1),  (2,1,1), 
(1,2,1),  (2,2,1)}  . 

a*  may  contain  several  unnecessary  states ,  states  whose  steady  state 
probability  will  be  zero  under  the  optimal  policy.  Although  Equations 
(7.7)  and  (7.13)  can  be  used  to  eliminate  some  of  these  states,  for 
small  examples  like  this  one,  it  is  probably  easier  to  carry  them 
along. 

Since  there  are  two  classes,  there  are  four  possible  actions, 
(0,0},  (1,0),  (0,1),  and  (1,1).  The  transition  matrix  for  each  action 
follows : 

To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 
(0,0,0)  -  --  --  --  - 

P(0,0)  - 

(1,1,0)  10000000 
(2,0,1)  10000000 
(1,2,0).  01000000 

■From 

State  (1,1,1)  00100000 

(2,1,1)  01000000 

(1,2,1)  00001000 

(2,2,1)  00010000 

A  row  of  dashes  indicates  that  the  action  cannot  be  chosen  when  the 
system  is  in  the  state  represented  by  the  row.  For  action  (0,0), 
the  next  transition  is  sure  to  be  the  completion  of  a  service. 
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To  State 

(0,0,0) (1,1,0) (2, 0,1' (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 


P(1,0)  - 


From 

State 


(0,0,0)  0 

(1,1,0)  0.67 

(2,0,1)  0.67 

(1,2,0)  - 
(1,1,1)  0 


0  0.67 


(2.1.1)  0  0.67 

(1.2.1)  - 
(2,2,1)  - 


0  0.33 


If  action  (1,0)  is  chosen  in  state  (0,0,0),  the  next  transition  is  sure 
to  be  the  joining  of  a  class  one  customer.  For  the  other  states  in 
which  action  (1,0)  may  be  chosen,  the  entries  are  the  result  of  the 
service  rate  competing  with  the  arrival  rate. 


To  State 


P(0,1) 


(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 

[o  0  1  0  0  0  0  o  1 


From 

State 


(0,0,0)  0  0 

(1,1,0)  0.5  0 

(2,0,1)  - 

(1,2,0)  0  0,5 

(1,1,1)  - 

(?,1,1)  - 

(1,2,1)  - 

(2,2,1)  - 


0.5  0 
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To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 


P(l,l)  - 


From 

State 


(0,0,0) 

(1,1,0) 

(2,0,1) 

(1,2,0) 

(1,1,1) 

(2,1,1) 

(1,2,1) 

(2,2,1) 


0  0.33  0.67  00000 

0.4  0  0  0.2  0.4  0  0  0 


If  action  (1,1)  is  chosen  in  state  (1,1,0),  the  arrival  rates  for  each 
class  compete  with  each  other  as  well  as  with  the  service  rate  to 
cause  the  next  transition. 

T_(k) ,  the  unconditional  expected  waiting  time  in  state  a  under 
_a 

action  k,  is  simply  the  reciprocal  of  the  transition  rate  out  of  state 
a  under  action  k.  Thus, 


*  \ 

A 

r 

A 

* 

A 

" 

- 

,  T (1,0)  - 

0.5 

,  T(0,1)  - 

0.25 

,  T(l,l)  - 

0.167 

0.25 

0.167 

0.125 

0.1 

0.25 

0.167 

- 

- 

0.25 

- 

0.125 

- 

0.25 

0.167 

- 

- 

0.25 

0.167 

- 

0.25 

’ 

- 

- 

.0-25, 

.  " 

.  -  . 

.-'■i  'XJ  i*.  ‘^ukk.1 


1!Z 


H  iA  v I.--"'  ■:'<!  1.JJ’ :  l  .  ''■> 
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The  components  of  all  vectors  for  this  example  correspond  to  the  states 
as  listed  in  a*;  that  is,  component  one  corresponds  to  state  (0,0,0), 
component  two  corresponds  to  state  (1,1,0),  and  so  on. 

If  b(0,0)  is  the  matrix  of  expected  rewards  for  transitions 
between  the  various  states  under  alternative  (0,0),  b(0,0)  is  the  null 
matrix  since  no  customers  are  allowed  to  join  under  alternative  (0,0). 

The  following  matrices  of  expected  rewards  for  transitions  under  the 
other  alternatives  were  found  using  Equation  (7.13). 

To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 

(0,0,0)  0  1.25  0  0  0  0  0  O' 

b(l,0)  - 

(1,1,0)  0  0  0  0.5  0  0  0  0 

(2,0,1)  0  0  0  0  0  0.5  0  0 

From  (1,2,0)  -------- 

State 

(1.1.1)  000000  -0.125  0 

(2.1.1)  0  0  0  0  0  0  0  -0.25 

(1.2.1)  -------- 

(2,2,1)  -  -  . 

If  the  only  nonzero  entries  in  a  row  of  b(k)  are  negative,  action  k  can 
be  eliminated  from  consideration  when  the  system  is  in  the  state  repre¬ 
sented  by  the  row  because  action  (0,0)  will  dominate  k  in  the  policy 
improvement  section  of  the  policy  iteration  algorithm. 


To  State 


(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 

(0,0,0)  f  0  0  0.375  0  0  0  0  0  ' 

b(0,l)  - 

(1,1,0)  0000  -0.25  000 

(2,0,1)  . 

(1,2,0)  000000  -0.875  0 

From 

State  (1,1,1)  -------- 

(2,1,1)  . 

(1,2,1) . . 

(2,2,1)  .  - 


To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (1,1,1) (2, 1,1) (1,2,1) (2, 2,1) 

(0,0,0)  0  1.25  0.375  0  0  0  0  0  ' 

b(l,l)  - 

(1,1,0)  0  0  0  0.5  -0.25  0  0  0 

(2,0,1)  . 

(1,2,0)  .  - 

From 

State  (1,1,1)  -------- 

(2,1,1)  .  - 

d.2,1) . 

(2,2,1)  -------- 
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where  a'  and  a"  are  used  to  denote  the  state  before  and  after  a 
transition,  respectively.  Recall  that  a*  is  a  set  that  contains  all 
states  that  are  required  for  the  social  optimum  problem.  For  this 
example. 


The  components  of  the  vector  of  expected  rewards  per  unit  time  in 
state  a  under  action  k  are  found  from 

q_(k)  -  r_(k)/T_(k) 

a  a  a 


The  optimal  policy  found  using  Algorithm  C.l  (Appendix  C)  is 

P*  -  (1,0)" 

(1,0) 

(1,0) 

(0,0) 

(0,0) 

(0,0) 

(0,0) 

.  (°>°)  . 

which  yields  g  »  1.714  and 

0  -  0.57 

0.29 
0.00 
0.14 
0.00 
0.00 
0.00 
.  °-00  . 

For  this  example,  it  is  socially  optimal  for  class  one  to  dominate  the 
system  to  the  exclusion  of  class  two .  Even  if  a  class  two  customer 
arrives  to  find  an  empty  system,  he  is  turned  away  since  the  adminis¬ 
trator  expects  the  customers  to  achieve  a  higher  gain  rate  by  leaving 


the  server  idle  until  a  class  one  customer  arrives. 


7.2  Class  Dependent  Service  Rate  Model 

The  assumptions  and  structure  of  this  model  are  the  same  as  for 

Model  II  except  each  class,  m,  has  its  own  service  rate,  y  .  For  an 

m 

arrival  to  be  able  to  compute  his  expected  time  in  the  system,  he  must 
know  the  number  of  each  class  of  customers  in  the  system.  As  illus¬ 
trated  in  Section  4.3,  this  requirement  together  with  a  first  come 
first  served  queue  discipline  leads  to  a  state  space  that  gives  the 

class  and  position  of  each  customer  in  the  system.  Thus,  the  state  of 

the  system  is  given  by  m  ■  (m^ ,  m^ ,  . . . ,  ,  . . . ) ,  where  m^  is  the 

class  of  the  customer  in  position  j .  The  customer  in  position  one  is 
in  service,  the  customer  in  position  two  is  first  in  line,  and  so  on. 

Self -optimizing  customers  of  class  m  will  join  the  system  if 
their  expected  net  benefit  for  joining  is  greater  than  zero  which  is 

their  expected  net  benefit  for  balking.  The  expected  net  benefit  of 

a  class  m  customer  joining  when  the  system  is  in  state  m  is 
M  _ 

R  -  ({  I  a,(m)/y  }  +  (1/y  ))c  ,  (7.14) 

m  ^  I  i  ra  '  m 


where  cr^(m)  is  the  number  of  class  i  customers  present  in  state  m. 
Since 


00 


I 

j-l 


(7.15) 


where 


6 


“  "J  ^  i 


(0  otherwise, 

a  class  m  customer  will  join  the  system  if  m  is  such  that  Equation 
(7.14)  is  nonnegative  or 
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l  {a  (m)/y  }  »  R  /C  “  i/h., 
li  mm  m 


(7.16) 


Since  Che  service  rate  varies  wich  customer  class,  Equation  (7.16) 

cannot  be  reduced  to  a  balking  point  in  terms  of  the  number  in  the 

system  like  n  .  Rather,  for  each  class  m,  let  S  be  the  set  of  all 
s  m 

m  _ 

m  such  that  Equation  (7.16)  holds.  Thus,  if  the  system  is  in  state  m, 

a  self-optimizing  customer  of  class  m  joins  if  meS^  and  balks  otherwise. 

It  will  be  argued  later  that  the  set  of  states  in  which  a  class 

m  customer  acting  in  a  socially  optimal  manner  joins  the  queue  is  a 

subset  of  the  set  of  states  in  which  he  would  join  if  he  were  acting 

in  an  individually  optimal  manner.  Thus,  if  0^  is  the  set  of  all 

states  such  that  customers  of  class  m  who  are  acting  in  a  socially 

optimal  manner  join  the  system,  then  0^  ^  S^.  A  little  additional  work 

must  be  done  to  bound  the  state  space  required  for  the  social  optimum 
M 

problem.  U  S  may  not  contain  enough  states  since  the  states  that 
i  m 
m*l 

can  be  reached  from  states  belonging  to  S  through  the  joining  of  a 

m 

class  m  customer  must  also  be  included  to  bound  the  social  optimum 

state  space.  Let  S'  contain  a)l  states  in  S  plus  all  states  that  can 

m  m 

be  reached  from  the  states  of  through  the  joining  of  a  class  m 

customer.  The  required  state  space  for  the  social  optimum  problem  can 

M 

then  be  bounded  by  U  S'. 

m-1 

Again,  a  self -optimizing  customer  fails  to  consider  the  decrease 
in  benefits  to  later  arriving  customers  caused  by  his  joining  the  queue. 
This  external  economic  effect  is  internalized  in  the  formulation  of  the 
social  optimum  problem  which  is 


max  max  7  „  rtP  P 

PeC  n  C  8P  "  PeC  0  C  -  M  ,  0-  q- 

sd  sdmeySmm 

ra-1 


(7.17) 
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Again,  only  stationary  Markovian  policies  are  considered  and  the  work 

of  Derman  (1962)  is  used  to  limit  the  search  to  a  nonrandomized  policy. 

P  — 

0_  is  the  steady  state  probability  of  the  system  occupying  state  m 

®  P 

under  policy  P.  q_  is  the  expected  gain  per  unit  time  the  system  is 
_  m 

in  state  m  while  policy  P  is  employed.  Equation  (7.17)  can  be 
rewritten  as 


PeC  nc,  Pec  AC 


s  d  -M  “  m-1 

®  U.Sm 

m-1 


0  T  k  (m)X  f  (m) 
m  u  ,  m  mm 


(7.18) 


where  k  (m)  is  one  if  class  m  is  admitted  under  policy  P  when  the 
in 

gvslem  is  in  state  m  and  Tero  if  not.  f  (m) ,  the  expected  system  gain 

m 

is  a  class  m  customer  is  admitted  when  the  system  occupies  state  m, 
fellows  from  Equation  (7.1+;  as 


f  (m)  -  R  -  (J  {a.(ra)/y  }  +  (1/y ))C  .  (7.19) 

m  m  j  i  l  .  1  m  '  m 


Although  the  state  space  is  quite  cumbersome,  the  seri-Markov  formula¬ 
tion  given  by  Equation  (7.18)  can  be  solved  by  policy  iteration. 

To  show  that  0  -  S  consider  first  the  quantity  t  ,  where 

mm  s 

m 

t  *  R  /C  -  1/y  .  S  contains  all  states  for  which  the  sum  of  the 
s  mm  mm 

m 

expected  service  times  of  all  customers  in  the  system  is  no  greater 

than  t  {see  Equation  (7.16)}.  If  t  is  defined  to  be  the  maximum 
s  o 

m  m 

amount  of  expected  service  time  in  the  system  for  which  a  social 

optimizing  customer  of  class  a  joins,  then,  t  *  t  is  analogous  to 

o  s 

<  m  m  < 
n  -  n  Theorem  4.3  can  be  adapted  to  show  that  t  -  t  Since 
os  Os 

mm  mm 

the  definition  of  0  implies  that  0  is  the  set  of  all  states  for  which 

m  m 

the  sum  of  the  expected  service  times  is  no  greater  than  t  ,  0  ^  S  . 

o  m  m 
m 


M wap 


*V,:  *•: 

. . 
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7.3  Nonpreemptive  Priority  Service  Discipline  Model  with  Class 
Dependent  Service  Rates 

This  model  is  a  combination  of  the  two  previous  models.  The 
assumptions  and  structure  are  the  same  as  Model  II  except  that: 

a)  a  nonpreemptive  priority  service  discipline  is  used,  and 

b)  the  mean  service  rate  varies  with  customer  class. 

Class  one  is  of  highest  priority,  class  two  of  next  highest,  and  so  on. 

Within  a  class,  the  service  discipline  is  first  come  first  served. 

Since  the  priorities  are  nonpreemptive,  the  customer  in  service  is 

allowed  to  finish  no  matter  what  the  priority  of  an  arrival.  The  mean 

service  rate  of  class  m  is  y  ,  m  -  1,  ...,  M.  Since  the  model  has  a 

m 

nonpreemptive  priority  service  discipline,  the  state  space  can  again 

be  a  vector  a,  where  a_  is  the  class  of  the  customer  in  service  and  a 

U  m 

is  the  number  of  class  m  customers  in  the  system. 

Self-optimizing  customers  of  class  m  will  join  the  system  if 
their  expected  net  benefit  for  joining  is  greater  than  their  expected 
net  benefit  for  balking  which  is  zero.  For  a  class  m  arrival,  the 
expected  net  benefit  of  joining  when  the  system  is  in  state  a  is 

R  -  t  (a)  C  ,  (7.20) 

mm  m 

where  t  (a)  is  the  time  the  arrival  expects  to  spend  in  the  system  if 
m 

he  joins  when  it  is  in  state  a.  Since  class  one  is  of  highest  priority, 
a  class  one  arrival  must  only  wait  for  the  customer  in  service  plus  any 
class  one  customers  already  in  the  system.  If  a  class  one  customer  is 
in  service,  he  should  not  be  counted  twice.  Thus,  the  time  a  class  one 
arrival  will  expect  to  spend  in  the  system  is  given  by 


t^Ca) 


where  6 


(a  *  i;/n  +  1  6  /v„ 

mm2  0 


1  if  aQ  ■  m 

0  otherwise 


(7.21) 


A  class  two  arrival  must  wait  for  all  class  one  and  class  two  customers 
present  in  the  system  plus  the  customer  in  service  if  he  has  not 
already  been  counted.  Thus, 

M 

t~(a)  -  a  /y  +  (a,  +  1)/V,  +  £  <5  o  /u  •  (7.22) 

2  112  2  m,art  m 

m— j  0 

In  general,  a  class  m  arrival  must  wait  for  all  customers  of  equal  or 
greater  priority  who  are  present  in  the  system  plus  the  customer  in 
service  if  he  has  not  already  been  counted.  Thus,  for  any  class  m, 
m-1  M 

t  (a)  -  l  a./y.  +  (am  4-  l)/u  +  \  6.  /y  .  (7.23) 

m  i-1  1  1  “  J-nri-1  j’a0  j 

Self-optimizing  customers  of  class  m  will  join  the  system  if 


R  ”  t  (a)C  -  R .  -  (l  a  /y  +  (a  4-  l)/y 

m  m  m  m  1  I  m  m 


+  l  <5,  /y.}C  ■  0 

j-m+1  ^a0  j  “ 


(7.24) 


Equation  (7.24)  can  be  written  as 


m  n 

(I  a  /y  +  l  5  /y.)  -  R  /C  -  1/y 

i-1  1  1  1-m+l  j’a0  J  “ 


(7.25) 


Since  the  mean  service  rate  varies  with  customer  class.  Equation  (7.25) 
cannot  be  reduced  to  a  balking  point  in  terms  of  customers  in  the 
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system  like  n  ;  however,  It  does  express  a  maximum  amount  of  expected 
s 

m 

service  time  that  can  be  ahead  of  a  class  m  arrival  and  still  have  him 


join  the  system,  t  *  R  /C  -  1/u  is  the  maximum  expected  time  a 
J  s  m  m  m 

m 

class  m  customer  can  wait  for  his  service  to  begin  and  still  find  it 


profitable  to  join  the  s>stem. 


Since  the  customers  are  served  on  the  basis  of  their  priority, 


the  expected  net  reward  for  a  class  m  member  of  the  queue  (i.e.,  in  the 


system  but  not  in  service)  is  altered  by  the  joining  of  a  customer  of 


higher  priority.  This  again  raises  the  problem  of  reneging  and  the 


various  assumptions  that  could  be  made  (see  Section  7.1). 


Let  S  be  the  set  of  all  states  such  that  Equation  (7.25)  holds, 
in 

It  will  be  argued  later  that  if  t  is  the  maximum  expected  time  a 

m 

class  m  arrival  can  wait  for  his  service  to  begin  and  still  find  it 

socially  optimal  to  join  the  system,  then,  t  ■  t  .  Thus,  if  0^  is 

m  m 

the  set  of  all  states  that  a  class  m  arrival  expects  to  wait  no  more 


than  t  for  his  service  to  begin,  then,  0  -  S  .  Let  S'  contain  the 

o  m  m  m 

m 

states  that  are  elements  of  S  together  with  the  states  that  can  be 


reached  from  the  elements  of  S  through  the  joining  of  a  class  m 
M  m 

customer.  U  S'  provides  a  bound  on  the  state  space  required  for  the 


social  optimum  problem. 


Self-optimizing  customers  ignore  the  decrease  in  net  benefits 


that  their  joining  causes  customers  of  lower  priority  either  present 


in  the  system  or  yet  to  arrive.  Both  of  these  costs  will  be  incorpor¬ 


ated  into  the  formulation  of  the  social  optimum  problem.  The  first 


will  be  charged  to  the  arrival  and  the  second  will  be  accounted  for  in 


the  objective  function  for  the  social  optimum  problem. 
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Consider  an  administrator  who  charges  a  joining  customer  for 
the  expected  costs  he  causes  other  customers  already  in  the  system  as 
well  as  for  his  own  expected  costs.  Since  the  joining  of  a  class  M 
arrival  affects  no  customers  present  in  the  system,  the  expected  gain 
in  state  a  due  to  his  joining  is 
M-l 

'  {(L  +  <aM  +  1)/yM>CM  ’  (7*26) 

The  joining  of  a  class  M-l  customer  affects  only  class  M  customers  who 
are  in  the  qusue.  Thus,  if  the  system  is  in  state  a,  the  expected 
system  gain  due  to  a  class  M-l  customer  joining  is 
M_2 

•m-1  -  ^  Vui  +  (Vl  +  UVl  +  4M,a0VCM-l 


<aM  "  SM,a0)CM/WM-l 


(7.27) 


In  general,  if  the  system  is  in  state  a,  the  expected  system  gain  due 
to  a  class  m  customer  joining  is 


Ml-  AT* 

l  -  {T  a  /y  +  (a  +  l)/y  +  V  5  /y.}C 

31  i-1  1  1  m  m  j-m+1  ^’a0  *  m 


M 

’  I  ^a4 
j-m+1  2 


(7.28) 


This  method  of  assigning  costs  avoids  reneging  since  a  customer's 
expected  cost  is  not  changed  by  later  arrivals  joining  the  queue. 
The  objective  function  for  the  social  optimum  problem  is 


max 

PeCMC. 
s  d 


gp 


max 

pec  ric, 

s  d 


l 


u  S '  a 
m-l  m 


a 


(7.29) 
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As  before,  only  nonrandomized ,  stationary  Markovian  policies  need  to 
P  P 

be  considered.  0_  and  q_  are,  respectively,  the  steady  state  probabil- 
a  _a 

ity  of  occupying  state  a  and  the  expected  gain  per  unit  time  in  state 
a  under  policy  P.  The  inclusion  of  the  steady  state  probabilities  of 
occupying  the  various  states  considers  the  expected  decrease  in 
benefits  to  later  arriving  customers  of  lower  priority  when  deciding 
whether  or  not  to  admit  a  customer.  Equation  (7.29)  can  be  written  as 


max 

Pecnc,  gP 

s  d 


PeCn  C  .  ^  M  E  .  km^a)Xmfm(a) 


8  d7'JS'  a  m"i 
m«l  m 


(7.30) 


where  again,  km(a)  is  one  if  class  m  is  admitted  under  policy  P  when 
the  system  is  in  state  a  and  zero  if  not.  f^(a) ,  the  expected  system 
gain  due  to  a  class  m  customer  joining  when  the  system  is  in  state  a, 
follows  from  Equation  (7.28)  as 


f  (a)  -  R  -  (I  a  /y  +  (a  +  l)/u  +  l  5.  /u, >C 

m  m  i-l  1  1  m  m  j-m+1  J’a0  d  ” 


-l  ,(aj  "  6j,a  )Cj/ym 
j=m+l  J  0  J 


(7.31) 


The  solution  of  the  semi-Markov  process  formulation  in  Equation  (7.30) 

by  policy  iteration  will  be  illustrated  by  an  example.  Theorem  4.3 

can  be  adapted  to  show  that,  for  all  m,  t  *  t  and  thus,  0  -  S  . 

os  mm 

m  m 

Consider  the  same  two  class  example  used  in  Section  7.1,  but 


let  each  class  have  its  own  service  rate.  The  mean  service  rates  and 


other  parameters  for  the  example  are  given  in  Table  7.1.  The  maximum 
expected  service  time  a  self-optimizing  customer  of  class  one  can  wait 
for  his  service  to  begin  and  still  join  the  system  is 
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t  -  R./C-  -  1/y,  -  0.344.  Similarly,  t  -  0.208.  Thus, 

^ 

Sx  -  {(0,0,0),  (1,1,0),  (2,0,1)},  and  S[  -  {(0,0,0),  (1,1,0),  (2,0,1), 
(1,2,0),  (2,1,1)}.  Also,  S2  -  {(0,0,0),  (2,0,1)},  and  S£  -  {(0,0,0), 
(2,0,1),  (2,0,2)}.  The  state  space  for  the  social  optimum  problem  is 
a  subset  of  y  S’  -  {(0,0,0),  (1,1,0),  (2,0,1),  (1,2,0),  (2,1,1), 


(2,0,2)}. 


U 

m-1 


m 


TABLE  7.1 

Parameters  for  the  Example 


Class 

R 

c 

X 

u 

m 

m 

m 

m 

1 

2 

3 

2 

3, 

.1 

2 

1 

2.5 

4 

5. 

.2 

The  transition  matrices  for  the 

four  possible 

actions  follow: 

To 

State 

(0,0,0)(1 

,1, 

0) (2,0,1) (1,2,0) (2 

,1,1) (2,0,2) 

(0,0,0) 

f 

— 

- 

— 

— 

> 

F(0,0) 

■1 

(1,1,0) 

1 

0 

0 

0 

0 

0 

(2,0,1) 

1 

0 

0 

0 

0 

0 

From 

State 

(1,2,0) 

0 

1 

0 

0 

0 

0  1 

(2,1,1) 

0 

1 

0 

0 

0 

0 

(2,0,2) 

0 

0 

1 

0 

0 

0  j 

Again,  a  row  of  dashes  indicates  the  action  cannot  be  chosen  when  the 
system  occupies  the  state  represented  by  the  row.  For  action  (0,0), 
the  next  transition  is  sure  to  be  the  completion  of  a  service. 
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To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (2, 1,1) (2, 0,2) 

(0,0,0) 

P(1,0)  - 

(1,1,0) 

(2,0,1) 

From 

State  (1,2,0) 

(2,1,1) 

(2,0,2) 

If  action  (1,0)  is  chosen  in  state  (0,0,0),  the  next  transition  is 
sure  to  be  the  joining  of  a  class  one  customer.  For  other  states  in 
which  the  action  may  be  chosen,  the  entries  are  the  result  of 
competition  between  the  rate  of  service  for  the  customer  in  service 
and  the  arrival  rate  of  class  one. 

To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (2, 1,1) (2, 0,2) 

(0,0,0)  f  0  0  1  0  0  0 

P(0,1)  - 

(1,1,0)  ------ 

(2,0,1)  0.57  0000  0.43 

From 

State  (1,2,0)  ------ 

(2,1,1)  . 

(2,0,2)  ------ 


161 


To  State 

(0,0,0) (1,1,0) (2,0,1) (1,2,0) (2, 1,1) (2, 0,2) 


(0,0,0) 

0  0.33  0.67  0  0  O' 

P(l,l)  - 

(1,1,0) 

------ 

(2,0,1) 

0.46  000  0.18  0.36 

From 

State  (1,2,0) 

------ 

(2,1,1) 

------ 

(2,0,2) 

.  4 

If  action  (1,1)  is  chosen  in  state  (2,0,1),  the  class  arrival  rate  of 
each  class  and  the  service  rate  of  the  customer  in  service  compete  to 
cause  the  next  transition. 

T_(k) ,  the  unconditional  expected  waiting  time  in  state  a  under 
_a  ___ 

action  k,  is  the  reciprocal  of  the  transition  rate  out  of  state  a  under 

action  k.  Thus, 


' 

,  T(1,0)  - 

0.5  ' 

,  T(0,01)  - 

0.25' 

,  T (1 ,1)  - 

0.17' 

0.32 

0.20 

- 

- 

0.19 

0.14 

0.11 

0.09 

0.32 

- 

- 

- 

0.19 

- 

- 

- 

0.19 

.  “ 

*  . 

- 

The  components  of  all  vectors  for  this  example  correspond  to  the 

2 

states  as  listed  in  y  S'. 


'  i.  i£t  A  t  ‘Li 


The  matrix  of  expected  rewards  for  transitions  between  states 
under  alternative  (0,0),  b(0,0),  is  the  null  matrix  since  no  customers 
may  join  the  system.  Equation  (7.31)  is  used  to  find  the  entries  in 
the  matrices  of  expected  rewards  for  transitions  under  the  other 
alternatives.  For  this  example, 

To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (2, 1,1) (2, 0,2) 


(0,0,0) 

0 

1.03 

0 

0  o 

0 

b(l,0)  - 

0.06  0 

0 

(1,1,0) 

0 

0 

0 

(2,1,0) 

0 

0 

0 

0  0.46 

0 

From 

State 

(1,2,0) 

— 

— 

(2,1,1) 

- 

- 

- 

- 

- 

(2,0,2) 

w  "" 

- 

- 

- 

4 

To  State 

(0,0,0) (1,1,0) (2, 0,1) (1,2,0) (2, 1,1) (2, 0,2) 


(0,0,0) 

0 

0 

0.52 

0 

0 

0 

b(0,l)  - 

(1,1,0) 

- 

- 

- 

- 

- 

- 

(2,0,1) 

0 

0 

0 

0 

0 

0.04 

From 

State 

(1,2,0) 

- 

- 

- 

- 

- 

- 

(2,1,1) 

- 

- 

- 

- 

- 

- 

(2,0,2) 

- 

- 

- 

- 

4 
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To  State 

(0,0,0) (1,1,0) (2,0,1) (1,2,0) (2,1,1) (2,0,2) 


(0,0,0) 

'  0  1.03  0.52  0  0  O' 

b(l,l)  - 

(1,1,0) 

------ 

(2,0,1) 

0000  0.46  0.04 

From 

State  (1,2,0) 

------ 

(2,1,1) 

------ 

(2,0,2) 

.  ~  “  “  “  “ 

The  components  of  r(k) ,  the  vector  of  expected  rewards  per 
transition  under  action  k,  are  found  from 


r_  (k)  -  7  2  P_  _  (k)b_  _  (k) 

a*  a"  Us*  aja"  aja" 

m-1  m 

where  a'  and  a"  are,  respectively,  the  state  before  and  after  a 
transition.  Here, 


*  ' 

,  r(l,0)  * 

1.03' 

,  r (0,1)  - 

0.52' 

,  r (1,1)  » 

0.69' 

0 

0.03 

- 

- 

0 

0.13 

0.02 

0.10 

0 

- 

- 

- 

0 

- 

- 

- 

.  0  J 

.  - 

.  ~  . 

k,  J 

The  components  of  the  vector  of  expected  rewards  per  unit  time  in 
state  a  under  action  k  are  found  using 

q_(k)  -  r_(k)/T_(k) 

a  a  a 
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Hence, 

q(0,0) 


The  optimal  policy  found  using  Algorithm  C.l  is  as  follows: 

P*  -  f (1,1)1 

(0,0) 

(0,0) 

(0,0) 

(0,0) 

.(0,0). 

which  yields  g  “  1.715  and  ^  ®  fo . 414] 

0.267 

0.319 

0.0 

0.0 

0.0 

For  this  example,  it  is  socially  optimal  for  class  one  and  class  two 
customers  tr  join  the  system  only  if  it  is  empty. 


7.4  Conclusion 

The  models  of  this  chapter  require  a  more  complex  state  space 


than  Models  I,  II,  and  III.  The  state  space  for  the  nonpreempti>/e 
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priority  models  is  tractable  as  lorg  as  the  number  of  classes  remains 
small.  However,  the  state  space  for  the  model  of  Section  7.2  is 
tractable  only  for  extremely  small  problems. 

The  thrust  of  this  chapter  has  been  to  demonstrate  that  the 
approach  used  in  Models  I,  II,  and  III  can  be  extended  to  other  models. 
That  is,  once  a  state  space  has  been  defined  that  carries  the  necessary 
information  for  memoryless  transitions  and  allows  computation  of  the 
expected  net  benefit  of  joining,  the  social  optimum  problem  can  be 
formulated  as  a  semi-Markov  decision  process.  Also,  the  individual 
optimum  problem  can  be  used  to  bound  the  number  of  states  required  for 
the  policy  iteration  solution  technique.  No  attempt  has  been  made  to 
draw  any  conclusions  about  the  form  of  the  optimal  policy. 


CHAPTER  VIII 


from  zero  to  the  expected  net  benefit  of  service  in  the  alternate 
system. ) 

The  semi-Marko”  approach  has  been  used  in  this  paper  to  extend 

the  model  vf  Naor  to  several  classes  of  customers.  Model  II.  For 

Model  II,  the  optimal  policy  for  social  optimizing  customers  was  shown 

to  Be  a  vector  of  forced  balking  points,  nQ.  Thus,  a  class  m  arrival 

joins  the  system  only  if  he  finds  the  number  of  customers  in  the 

system  to  be  less  than  n  ,  the  balking  point  for  his  class.  The 

m 

optimal  policy  for  self-optimizing  customers  was  also  shown  to  be  a 

vector  cf  balking  points,  n  .  Finally,  it  was  shown  that  the  social 

s 

and  self-optimum  balking  points  for  each  class  are  related  by  n  *  n 

m 

which  facilitates  solution  of  the  social  optimum  problem  by  providing 
a  bound  on  the  required  state  space.  Both  a  policy  iteration  and  a 
linear  programming  approach  were  presented  as  solutions  of  Model  II. 

The  semi-Markov  formulation  of  Model  II  was  generalized  in 
Chapter  V  to  Model  III  which  incorporates  Erlang  service  times.  This 
generalization  enhances  :he  usefulness  of  the  model  by  providing  more 
flexibility  in  representing  the  distribution  of  service  times.  The 
Erla~ g  problem  was  solved  by  employing  the  method  of  phases  which 
replaces  a  single  Erlang  h  service  time  with  the  mathematically 
equivalent  sum  of  h  independent,  identical.1  v  distributed,  exponential 
service  times.  Although  this  approach  enables  the  policy  iteration 
and  linear  programming  solution  techniques  of  Model  II  to  be  applied 
to  I.odel  III,  it  can  lead  to  an  optimal  poll.cy  that  forces  the 
administrator  to  identify  the  various  phases  of  service.  Since,  in 
general,  the  phases  are  not  physical  attributes  of  the  system,  the 
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solution  found  may  not  be  implementable.  A  heuristic  procedure  was 
suggested  for  determining  a  good  solution  that  does  not  require  the 
identification  of  the  phases  of  service.  In  addition,  a  mixed  integer 
programming  formulation  was  presented  that,  if  solved,  would  identify 
the  optimal  implementable  solution.  The  set  of  states  required  for 
the  social  optimum  problem  was  again  shown  to  be  a  subset  of  the  set 
of  states  required  for  the  individual  optimum  problem. 

In  Chapter  VI,  Models  II  and  III  were  applied  to  the  landing 
queue  at  the  Greater  Pittsburgh  International  Airport.  Data  available 
from  the  FAA  and  other  easily  accessible  sources  were  used  to  specify 
the  required  parameters  for  five  classes  of  customers.  Data  taken  at 
the  Pittsburgh  Airport  led  to  a  rather  cumbersome  Erlang  19  distribu¬ 
tion  of  service  times.  This  pointed  out  that  Model  III  can  create 
difficulties  for  both  the  policy  iteration  and  mixed  integer  pro¬ 
gramming  solution  methods  by  requiring  a  large  number  of  states. 
Although  special  techniques  for  solving  such  large  problems  exist, 
this  particular  problem  was  solved  by  bounding  the  Erlang  19  results 
between  the  results  of  an  Erlang  8  model  and  those  of  a  deterministic 
service  time  model.  The  deterministic  service  time  model  used  is  an 
approximate  model  based  on  the  work  of  Adler  and  Naor  (1969) . 

The  semi-Markov  approach  was  extended  to  three  new  models  in 
Chapter  VII  to  illustrate  the  versatility  of  the  approach.  The 
nonpreemptive  priority  discipline  models  (one  with  and  one  without 
class  dependent  service  rates)  require  a  state  space  that  gives  the 
number  of  each  class  of  customers  in  the  system  together  with  the 
class  of  the  customer  in  service.  This  new  state  space  requires  more 
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states  than  that  of  Model  II  for  the  same  size  problem  and  thus, 
will  tend  to  cause  more  computational  difficulties  for  both  the 
policy  iteration  and  linear  programming  solution  technique.  The  other 
model,  a  first  come,  first  served  model  with  class  dependent  service 
rates,  requires  a  state  space  that  specifies  the  class  of  the 
customer  in  every  position  in  the  system.  This  state  space  compounds 
the  computational  problems  even  further.  For  all  three  models,  the 
state  space  required  for  the  social  optimum  problem  is  again  a  subset 
of  the  state  space  required  for  the  individual  optimum  problem. 

The  semi-Markov  decision  process  formulation  used  in  this  paper 
applies  to  a  broad  range  of  queueing  control  problems  as  illustrated 
by  the  variety  of  models  solved  in  this  paper.  The  semi-Markov 
formulation  of  Models  II  and  III  and  each  of  the  models  in  Chapter  VII 
can  be  solved  using  a  technique  based  on  policy  iteration  or  linear 
programming.  The  solution  technique  may  have  to  be  tailored  to  the 
problem  as  was  the  case  with  Model  III.  When  a  large  number  of  states 
is  involved,  the  solution  technique  may  encounter  computational  prob¬ 
lems;  however,  special  techniques  such  as  the  bounding  approach  of 
Chapter  VI  can  often  be  applied  to  get  useful  results  even  in  these 
cases. 

For  all  models  considered,  the  social  optimum  balking  point  (in 
terms  of  customers  or  expected  service  time  in  the  system)  for  each 
class,  m,  is  m  greater  than  the  self-optimum  balking  point.  This 
characteristic  of  the  models  Is  useful  in  implementing  the  solution 
techniques  for  the  social  optimum  problem  since  it  provides  a  bound 
on  the  state  space  required  for  the  social  optimum  problem.  The 
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control-limit  property  of  the  optimal  solution  to  Model  I  carries  over 
to  Model  II.  The  form  of  the  optimal  solution  was  not  investigated 
thoroughly  for  the  other  models . 

Models  II  and  III  can  be  successfully  applied  to  the  problem  of 
deciding  which  classes  of  aircraft  to  admit  to  the  landing  queue  of  an 
airport.  The  results  are  useful  in  planning  the  schedule  of  arrivals 
at  the  airport. 

8 . 2  Potential  Uses  of  the  Models 

In  Chapter  VI,  Models  II  and  III  were  used  to  determine  an 
optimal  policy  for  admitting  aircraft  .o  the  landing  queue  of  an 
airport.  The  models  can  be  applied  to  other  similar  problems.  For 
instance,  the  service  might  be  the  unloading  of  an  oil  tanker  (or  a 
merchant  ship  or  a  truck) .  The  tankers  could  be  broken  into  classes 
based  on  capacity  or  type  of  crude  oil.  A  policy  of  admitting  tankers 
to  the  queue  at  the  port  would  be  sought  so  that  the  gain  rate  of  all 
tankers  acting  as  a  group  is  maximized. 

Systems  modeled  need  not  have  customers  that  physically  line 
up.  Consider  the  problem  of  using  a  single  train  or  truck  fleet  to 
haul  grain  from  elevators  throughout  Nebraska  to  Kansas  City.  In  this 
problem,  the  server  moves  to  the  customers  and  the  queue  is  a  schedule 
of  elevators  the  train  is  committed  to  serve.  The  classes  of  customers 
might  be  based  on  elevator  capacity,  location,  or  type  of  grain.  The 
problem  remains  that  of  determining  a  policy  of  admitting  customers  to 
the  queue  (list)  so  that  the  gain  rate  of  all  customers  taken  as  a 
whole  is  maximized. 

Some  problems  that  cannot  be  solved  directly  using  Models  II 
and  III  can  be  solved  by  combining  the  models  with  other  techniques. 
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For  instance.  Chapter  VI  suggested  combining  Model  II  or  III  with  a 
search  technique  to  determine  a  schedule  of  arrivals  at  an  airport, 
which  achieves  an  acceptable  gain  rate  for  the  customers  while  keeping 
the  probability  of  rejection  for  each  class  below  some  specified  bound. 

A  slightly  different  example  is  the  problem  of  choosing  the 
number  of  computer  terminals  allowed  to  tie  into  a  computer  system. 

The  classes  of  customers  might  be  based  in  type  of  user  account  or  on 
type  of  service  demanded.  Choosing  the  number  of  terminals  allowed 
for  each  class  does  not  directly  determine  queue  size  but  rather 
determines  the  size  of  the  population  of  each  class  of  customers.  If 
the  class  arrival  rates  can  be  determined  as  a  function  of  class 
population,  then  a  search  technique  could  be  used  in  combination  with 
Model  II  or  III  to  find  a  socially  good  policy  for  allowing  terminals 
to  tie  into  a  computer  system. 

Still  another  similar  application  is  that  of  choosing  the  number 
of  skiers  to  admit  to  a  ski  resort  on  a  hclidy  or  weekend.  The 
customers  might  be  classified  by  skiing  ability  which  would  indicate 
which  parts  of  the  ski  area  they  would  be  likely  to  use.  The  service 
demanded  is  the  use  of  a  ski  lift  to  carry  the  skier  to  the  top  of 
the  slope.  Since  most  ski  areas  have  more  than  one  lift,  class 
arrival  rates  must  be  determined  for  each  lift,  as  a  function  of 
population.  Again,  a  search  tehcnique  could  be  combined  with  Model 
II  or  III  to  find  socially  good  policy  of  admitting  skiers  to  the  ski 


area. 
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8.3  Suggestions  for  Further  Research 

The  work  described  in  this  paper  can  be  carried  forward  in 
several  different  directions.  The  heuristic  procedure  introduced  to 
provide  a  good  solution  to  Model  Ill  assumes  that  the  optimal  policy  is 
a  control-limit  policy  for  each  class.  The  assumption  is  based  on  the 
fact  that  the  expected  net  benefit  for  joining  decreases  as  the 
number  in  the  system  increases.  Implementing  a  policy  that  is  not 
a  control-limit  policy  would  force  a  customer  to  balk  when  i  are  in 
the  system  but  allow  him  to  join  when  i  +  1  are  in  the  system  which 
yields  the  customer  a  smaller  expected  net  benefit  than  i.  Although 
this  argument  supports  the  control-limit  assumption,  a  proof  of  the 
control-limit  property  would  lend  more  credibility  to  the  heuristic 
procedure. 

The  expected  net  benefit  of  joining  for  customers  in  the  first 
come,  first  served,  class  dependent  service  rate  model  of  Chapter  VII 
decreases  as  the  expected  service  rime  ahead  of  the  arrival  increases. 
Arguments  similar  to  those  used  above  for  Model  III  can  also  be  used 
to  support  the  assumption  that  the  optimal  policy  for  each  class 
establishes  a  control  limit  in  terms  of  the  expected  service  time 
ahead  of  an  arrival.  The  nonpreemptive  priority  models  of  Chapter  VII 
charge  an  arrival  for  both  his  own  expected  costs  and  the  increase  in 
expected  costs  he  causes  customers  of  lower  priority  already  present 
in  the  system.  This  complicates  the  definition  of  a  control  limit 
even  further  since  the  expected  net  benefit  of  joining  decreases  with 
an  increase  in  the  number  of  customers  of  lower  priority  in  the  system. 
The  form  of  the  optimal  policy  needs  to  be  investigated  further  for 
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each  of  the  three  models  of  Chapter  VII.  In  addition,  use  of  the 
three  models  would  be  facilitated  by  adapting  the  program  of  Appendix 
D  to  handle  them. 

On  a  broader  scope,  additional  work  should  be  doite  to  determine 
the  range  or  variety  of  models  to  which  the  semi-Markov  approach 
applies.  Subsets  of  these  models  should  be  sought  for  which  the 
social  optimum  state  space  is  contained  in  the  individual  optimum 
state  space.  Also,  the  subsets  should  be  identified  for  which  a 
control-limit  policy  is  optimal.  Further,  the  results  of  the  models 
of  this  paper  should  be  compared  with  results  of  other  similar  models 
such  as  those  of  Balachandran  and  Schaefer  (1975,  1976)  that  use. 
expected  queue  length  rather  than  number  of  customers  in  the  system 
to  determine  the  expected  costs  of  joining. 

Some  refinements  and  extensions  that  have  been  applied  to 
Model  I  can  also  be  applied  to  Models  II  and  III.  For  example,  tolls 
similar  to  those  used  in  Model  I  to  implement  the  socially  optimal 
policy  should  be  developed  for  Model  II  by  using  Equation  (3-25)  to 
find  a  toll  for  each  class  of  customers.  Also,  the  approaches  used 
by  Knudsen  (1972)  and  Yechiali  (1972)  to  extend  single  class  models 
to  include  multiple  servers  could  be  tried  on  the  multiclass  models 
as  well. 

Finally,  the  work  presented  in  this  paper  has  touched  several 
other  interesting  problems  that  should  be  investigated.  One  such 
problem  is  the  expansion  of  the  deterministic  service  time  model  of 
Adler  and  Naor  (1969)  to  several  classes  of  customers.  Another  is 
the  development  of  a  model  in  which  the  administrator  can  control 


customer  arrival  rates  through  population  size  rather  than  queue  size. 
One  final  problem  is  the  use  of  objective  functions  other  than  the 
individual  and  social  optimum  ones  used  in  this  paper.  One  such 
objective  function  used  by  Naor  (1969)  assumes  that  the  administrator 
is  trying  to  maximize  the  sum  of  the  tolls  he  collects  from  joining 
customers. 

Certainly,  many  other  areas  for  further  research  exist. 

However,  these  few  are  listed  to  give  the  researcher  some  idea  as 
to  where  he  might  start. 
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APPENDIX  A 

GLOSSARY  OF  NOTATION 


This  glossary  defines  notation  used  throughout  this  dissertation. 
Mathematical  symbols  and  operators,  such  as  the  greatest  integer 
function  [..]  ,  are  not  included.  Also  missing  is  some  notation  that 
is  used  only  locally  for  one  idea  or  development.  The  symbols  are 
listed  in  alphabetical  order  with  Greek  symbols  appearing  at  the  end 
of  the  list  for  their  English  equivalent;  for  example,  p  is  listed  at 
the  end  of  the  r ' s . 

a  (a^,  a^,  ...,  a^) ,  state  space  for  the  nonpre- 

emptive  priority  models,  where  a^  is  the  class 

of  the  customer  in  service  and  a  is  the  number 

m 

of  class  m  customers  in  the  system 
B  Large  numerical  value  required  to  develop  the 

either /or  constraints  in  the  mixed  integer 


bi3 (k> 
C 


^dc£ 


C 

m 


program 

Expected  reward  for  a  transition  from  state  i 
to  j  when  action  k  is  chosen 

Cost  per  unit  time  in  the  system  for  customers 

in  a  single  class  problem 

Class  of  control-limit  policies 

Class  of  nonrandomized  policies 

Class  of  deterministic  control-limit  policies 

Cost  per  unit  time  in  the  system  for  customers 


of  class  m 
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|Bew' 


wt 

d(h) 


Di(k) 


D*(k) 


<(Hm- 


Class  of  stationary  Markovian  policies 
Class  of  all  policies 

Policy  chosen  in  iteration  h  of  the  policy 
iteration  altorithm 

Stationary  probability  that  action  k  is 
chosen  when  the  system  is  in  state  i 
Stationary  probability  that  action  k  is 
chosen  under  policy  P  when  the  system  is  in 
state  i 

,  ,r,  )  Probability  that  action  k  is  chosen  under 
1  m 

policy  P  at  time  m  given  history  and 


n 


f  (i) 
m 


present  state 


th 


Decision  made  by  the  n  arrival  (A  *  O') 
if  balk,  1  if  join) 

Number  of  customers  in  the  system  at  the 
instant  of  the  n^  arrival 

Expected  net  reward  to  a  joining  customer  of 
class  m  if  i  customers  are  in  the  system 
Expected  net  benefit  rate  or  gain  rate  of  the 
system 

Expected  gain  rate  using  the  formulation  given 
by  Naor  (1969) 

Expected  gain  rate  of  the  system  under  policy  P 
Expected  gain  rate  using  the  formulation  which 
delivers  the  reward  when  a  customer  joins  the 
system  but  distributes  the  charges  for  time  in 
the  system  throughout  the  customer's  stay 


.i1  Law 


ie  M. 


180 


L 

1® 

X 

X 

m 

X' 


X' 

m 

Mi) 


Mi) 

m 


M 

ra 


Expected  gain  rate  using  the.  formulation  given 
by  Yechiali  (1971) 

Erlang  number 

History  of  the  process  through  arrival  m 
(k^,  k^,  k^)  ,  a  possible  action,  where 

k  *■  0  if  class  m  balks  and  1  if  it  joins 
Indicator  which  is  zero  if  class  m  balks  when 
the  state  of  the  system  is  i  and  one  if  class  m 
joins  when  the  state  is  i 
Expected  number  of  customers  in  the  system. 
Contribution  of  class  m  to  the  expected  number 
of  customers  in  the  system 

Arrival  rate  of  customers  in  the  single  class 
problem 

Arrival  rate  of  class  m  customers 

Effective  arrival  rate  of  customers  in  the.  sing" 

class  problem 

Effective  arrival  rate  of  class  m  customers 
Arrival  rate  of  customers  in  the  single  class 
problem  when  the  system  is  in  state  i 
Arrival  rate  of  class  m  customers  when  the 


system  is  in  state  i 

Number  of  classes  of  customers 


(ra^,  m^j  .  .  . ,  m  ,  ...),  state  space  which  gives 
the  position  of  customers  in  the  system,  where 


m  indicates  the  class  of  the  customer  occupying 
position  j 


Service  rate  of  the  single  server 

Service  rate  for  class  m  customers  in  the  class 


dependent  service  rate  models 

(n^,  n^,  ...»  n^) ,  vector  of  balking  points 


(n  ,  n  ,  . . . ,  n  ) ,  vector  of  forced  balking 
°1  °2  °M 

points  for  the  social  optimum  problem 


(n  ,  n  ,  . . . ,  n  ) ,  vector  of  balking  points 
S1  s2 

for  the  individual  optimum  problem 
Largest  component  of  n 

s 

Set  of  states  for  which  a  social  optimizing 
arrival  from  class  m  joins  the  system 
Policy  for  controlling  the  system 
Probability  of  a  transition  from  state  i  to  j 


under  action  k 


Steady  state  probability  that  i  customers  are 
in  the  system  under  policy  P 
Fixed  payment  to  an  arrival  who  does  not  join 
the  system 

Amount  paid  per  customer  in  the  system  to  an 
arrival  who  does  not  join  the  system 
Expected  net  reward  per  unit  time  in  state  i 
under  action  k 


Reward  for  service  of  a  customer  in  the  single 
class  problem 

Expected  reward  par  occupancy  of  state  i  under 


action  k 
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R 


m 


RST 


P 

S 


S 


k 


S 

m 


S' 

m 


CT  (m) 

m 

T 


t  (  a) 

IQ 


T(k) 


Reward  for  service  of  a  member  of  class  m 
Remaining  service  time  for  the  customer  in 
service 

X/u,  the  traffic  intensity 

Class  of  stationary  control-limit  policies  of 
infinite  order 

Class  of  stationary  control-limit  policies  of 
order  k 

Set  of  states  for  which  a  self -optimizing  arrival 
from  class  m  joins  the  system 
State  space  required  for  class  m  in  the 
individual  optimum  problem 

Number  of  customers  of  class  m  present  in  state  m 
Length  of  a  service  for  the  deterministic  model 
Transition  rate  out  of  state  i 
Expected  time  in  the  system  for  a  class  m 
arrival  who  finds  the  system  in  state  a 
Maximum  amount  of  expected  service  time  ahead  of 
a  social  optimizing  class  m  customer  for  which 
he  will  join  the  system 

Maximum  amount  of  expected  service  time  ahead  of 
a  self-optimizing  class  m  customer  for  which  he 
will  join  the  system 

Vector  of  unconditional  expected  waiting  times 
under  action  k 


X 


Unconditional  expected  waiting 
under  action  k 
Expected  holding  time  in  state  i  under  action  k 
given  the  next  transition  is  to  state  j 
Relative  value  of  the  systan  starting  in  state  i 
Continuous  variable  representing  the  forced 
balking  point  for  the  single  class  social 
optimum  problem 

Continuous  variable  representing  the  balking 
point  for  the  single  class  self -optimum  problem 
Cost  per  unit  time  of  operating  the  system  in 
state  i 

0i/ti<k),  decision  variable  in  the  linear 
programming  formulation  of  a  semi-Markov 
decision  process 

Binary  variable  that  indicates  whether  or  not 
action  k  is  chosen  when  i  customers  are  in  the 
system  (used  in  the  mixed  integer  programming 
formulation  of  the  Erlang  service  time  problem) 


APPENDIX  B 


PROOFS  OF  SOME  ASSERTIONS  ABOUT  MODEL  I 

Naor  (1969)  states  some  properties  of  Model  I  but  does  not 
provide  supporting  proofs.  The  following  three  properties  are 
established  in  this  appendix: 

1)  g(n),  the  socially  optimal  gain  rate  is  discretely 
unlmodal  in  n,  the  forced  balking  point. 

2)  v  increases  with  v  ,  where  v  ■  Ry/C  and  v  and 

'  s  o’  s  o 

v  are  related  by  Equation  (3.5). 
s 

3)  v  ■  v  ,  where  again  v  and  v  are  related  by 

os  o  s 

Equation  (3.5). 

B.l  Proof  of  Property  One 

Property  (1)  is  important  because  it  guarantees  that  a  local 
optimum  is  a  global  optimum. 

Theorem  B.l:  g(n)  is  discretely  uniaodal  in  n. 

Proof:  Define  Ag(n)  to  be  g(n)  -  g(n  -  1).  Since 
R  ■  C/y,  g(l)  ■  0.  This  together  with  g(0)  »  0 
implies  that  Ag(l)  ■  0.  Thus,  to  establish  the 
theoremr  it  is  sufficient  to  show  that  if  Ag(n)  -  0, 
then  Ag(n  +  I)  <  0.  Equation  (3.3)  is  provided  below 
for  convenience. 

Ag(n)  -  XR{1  -  Pn(l  -  P)/(l  -  Pa+1)} 

-  C(p/(1  -  P)  -  (n  +  1)PQ+1/(1  -  Pn+1)} 


* 
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Use  of  Equation  (3.3)  for  g(n)  and  g(n  -  1)  yields 

g(n)  -  XR{pn'1(l  -  p)/(l  -  Pn)  -  Pn(l  -  P)/(X-  Pn+1)> 

-  C(npn/(1  -  Pn)  -  (n  +  l)pn+1/d  -  Pn+1)> 

-  xspn_:L(i  -  p)2/{(i  -  Pn)d  -  Pn+1)> 

-  Cpn{n(l  -  p)  -  p  +  Pn+1>/{(1  -  P°)<1  -  Pn+1)> 

(B.l) 


Similarly, 

Ag(n  +  1)  -  XRpn(l  -  p)2/((l  -  PQ+1)(1  -  Pn+2)} 

-  Cpn+1((n  +  1)(1  -  p)  -  p  +  Pn+2)/ 

{(1  -  pn+1)(l  -  Pn+Z)>  •  (B.2) 

With  the  addition  and  subtraction  of 
c^/Ui  -  pa+1)U  -  Pn+2)1, 

Equation  (B.2)  can  be  written  *s 

Ag(n  +  1)  -  (XRpn(l  -  P)2  -  Cnpn+1(l  -  p) 

+  cPn+2  -  cp2n+2}/{(i  -  Pn+1)d  -  Pn+2>) 
+  {-cp^+1d  -  p)  +  cp2n42  -  cp2n+3>/ 

{(1  -  pn+1)(l  -  P^2)}  •  (B.3) 


Multiplication  of  Equation  (B.3)  by 

p(l  -  pQ)(l  -  pn+2)/(p(l  -  PQ)(1  -  pn+2)> 


yields 
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Ag(n  +  1)  -  ( (XRpn-1(l  -  p)2  -  Cnpn(l  -  p) 

+  cPn+1  -  cp2n+1)/{(i  -  Pn)(i  -  pn+1)}) 

x  {pa  -  Pn)/(i  -  Pn+2)} 

+  cpn+1(i  -  p) (p®*’1  -  1)/ 

{(1  -  pn+1)(l  -  pn+2)}  .  (B.4) 

Equation  (B.4)  can  be  written  as 

Ag(n  +  1)  -  Ag(n)p(l  -  pn)/(l  -  pn+2) 

-  Cpn+1(l  -  p)/(l  -  pn+2)  .  (B.5) 

For  p  >  0,  p(l  -  pn)/(l  -  pn+2)  >  0 

and  Cpn+1(l  -  p) / (1  -  pn+2)  >  0.  Thus,  if  Ag(n)  -  0, 

Equation  (B.5)  yields  Ag(n  +  1)  <  0  and  establishes 
the  theorem. 

B.2  Proof  of  Property  Two 

vg  and  vq  are  continuous  variables  that  satisfy  the  equalities 

in  the  formulas  that  define  ng  and  nQ,  respectively.  Naor  (1969) 

shows  that  n  ■  fv  1 ,  where  the  brackets  denote  the  greatest  integer 
s  L  s 

in  v  .  If  Property  (2)  holds,  that  is,  if  vg  increases  as  vq  increases, 
then  nQ  ■  [v  ]  specifics  the  integer  that  satisfies  Equation  (3.4)  and 
is  thus,  the  balking  point  for  the  social  optimum  problem. 

One  relationship  (given  by  Naor)  that  is  used  in  the  proof  of 
Property  (  .)  is  vq  ■  1.  This  relationship  arises  from  Equation  (3.5) 
which  is 

v  , 

{v  (1  -  p)  -  p(l  —  p  )}/(!  —  p)  ■  v 
o  s 
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Substitution  of  v  ■  1  into  Equation  (3.5)  yields  v  -1.  Substitution 
o  s 

of  vq  <  1  yields,  v  <  1,  but  this  leads  co  a  trivial  system  since 

n  *  0.  Since  trivial  systems  are  avoided  by  requiring  that  R  -  C/y, 
s 

the  relationship  is  validated. 


Theorem  B.2:  3v  /3v  >0. 

-  s  Q 

Proof:  The  proof  of  this  theorem  is  divided  up  into 
proofs  for  each  of  three  cases, 

1)  0  <  p  <  1, 

2)  P  -  1, 

3)  p  >  1. 

Case  1.  0  <  p  <  1. 

First,  Equation  (3.5)  can  be  written  as 

v  +1 

vo  -  v /<i  -  p)  -  p/u  -  p)2  +  p  °  /a  -  p)2. 

3  U 

Thus, 

7  v  +1 

3v  /3v  -  1/(1  -  p)  +  {1/(1  -  p)  )(3p  °  /3v  ) 

so  o 

v  +1  2 

-  1/(1  -  p)  +  Jlnp  p  0  /(I  -  p)Z.  (B.6) 

Since  v  »  1  and  Jlnp  <  0, 
o 

3v  /3v  -  1/(1  -  p)  +  p2inp/(l  -  p)2 

s  o 

-  (1  -  p  +  p2Jlnp)/(l  -  p)2  . 

2 

Since  (1  -  p)  >  0,  it  must  be  shown  that 
2 

1  -  p  +  p  Jlnp  >  0  .  (B .  7) 
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Equation  (B.7)  can  be  written  as 

(p  -  l)/(p2inp)  >1.  (B.8) 

First,  examine  Equation  (B.8)  at  the  end  points  of 
the  region;  that  is,  at  p  ■  0  and  p  -  1. 

JJJJ  t(P  -  l)/(p2*np))  -  J^(Up  -  l)/pVnp) 

which  is  indeterminate.  Application  of  L'Hospital's 
rule  yields 

-  1)  / p2}/&np)  -  (2  -  p)/p2 

which  is  undefined  (+°°)  .  Also, 

^  {(p  -  1)7 (p2&np) }  -  {l/(2pinp  +  p)}  -  1, 

with  the  help  of  L' Hospital's  rule.  Thus,  if 

d{ (p  -  l)/(p2inp) }/dp  <  0  for  0  <  p  <  1, 

then,  Equation  (B.8;  holds  and  the  theorem  is  established 
for  Case  1.  Differentiation  yields 

d{(p  -  l)/(p2&np) }/dp  *  {p2£np  -  (p  -  l)(2p&np  +  p)}/ 

2  2 
(p  «apr 

■  (pJlnpCl  -  p)  + 

+  p(£np  -  p  +  l))/(p2£np)2. 

Since  the  denominator  is  greater  than  zero,  the 
numerator  must  be  shown  to  be  less  than  zero. 


For  0  <  p  <  1, 
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pJtnp(l  -  p)  <  0  . 

Also,  from  the  series  expansion  of  £np, 
inp  <  p  -  1  . 

Thus, 

p(£np  -  p  +  1)  <  p(p  -  1  -  p  +  1)  -  0. 

This  demonstrates  that 

d { (p  -  l)/(p^Anp) }/dp  <  0  for  0  <  p  <  1 
and  establishes  the  theorem  for  Case  1. 


Case  2,  p  *  1. 

Substitution  of  p  -  1  into  Equation  (3.5)  yields  an 
indeterminate  form.  Evaluation  of  the  limit  as  p 
approaches  ore  of  Equation  (3.5)  is  somewhat  complicated 
by  the  fact  that  vq  is  a  function  of  p.  However,  two 
applications  of  L'Hospital's  rule  yield 


(B.9) 


(B .  10) 
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Theorem  B.3:  v  ■  v  . 

-  o  s 


Proof:  Theorem  3.1  establishes  that  n  and  n  in  Naor 

o  s 

(1969)  are  the  same  as  in  Yechiali  (1971) .  To  prove  the 

theorem,  assume  v  <  v  .  Since  n  ■  fv  1  and  n  *  v  , 
so  o  1  oJ  s  L  sJ 

n  ■  n  .  This  contradicts  Yechiali' s  result  and  establishes 
o  s 

the  theorem. 


B.4 


Demonstration  of  Equation  (3.5) 

Use  of  Equation  (3.5)  is  demonstrated  on  the  one  class  example 


pi  i'' em  of  Table  3.1  with  X  ■  1,  The  other  parameters  of  the  modal  are 


R  »  5,  r-“2,U«3,  p  ■  1/3.  v  ■  Ry/C  ■  7.5.  Substitution  of  this 

s 

result  '.nto  Equation  (3.5)  yields 


7.5  -  (vq(2/3)  -  (1/3) (l  -  (1/3)  °}) / (4/9) 
or 

v  +1 

(2/3)vq  +  (1/3)  0  -  11/3  . 


vq  ■  5  yields  3.3347  which  is  too  small,  but  vq  ■  6  yields  4.0005 

which  is  too  large.  Since  v  lies  between  5  and  6,  n  “5  which 
'*o  o 

agrees  with  Table  3.1. 
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INTRODUCTION  TO  HARKOV  DECISION  PROCESS 

This  appendix  is  a  brief  introduction  to  Harkov  and  semi-Harkov 
decision  processes  including  the  use  of  the  policy  iteration  algorithm. 
The  reader  who  desires  a  more  thorough  development  is  referred  to 
Howard  (1971)  and  Ross  (1970) . 

C . 1  Markov  Decision  Processes 

The  first  process  considered  is  a  discrete  time,  finite  horizon, 
finite  state  space,  undiscounted,  Harkov  decision  process.  A  Markov 
process  can  occupy  any  of  a  number  of  states.  In  this  paper,  the 
states  usually  correspond  to  the  number  of  customers  in  the  system. 

Some  mechanism  causes  the  process  to  jump  from  one  state  to  another 
according  to  a  probability  distribution.  The  jumps  or  transitions  are 
equally  spaced  in  time.  The  process  is  Markovian  because  knowing  the 
present  state  of  the  process  is  as  good  as  knowing  the  entire  history 
of  the  process  when  trying  to  predict  the  next  state.  The  process  is 
a  decision  process  because  at  each  transition,  an  administrator  can 
choose  an  action  from  among  a  set  of  actions.  The  administrator’s 
choice  affects  the  probability  distribution  of  the  next  transition. 

Since  the  transitions  are  equally  space.-’  in  time,  the  process  is  called 
a  discrete  time  process.  The  process  has  a  finite  horizon  because  it 
stops  at  some  given  point  of  time  in  the  future.  In  this  paper,  a 
bound  exists  for  the  maximum  number  of  states  required  for  the  process; 
thus,  the  process  has  ?  finite  state  space.  The  process  is  undiscounted 
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because  it  is  assumed  that  a  dollar  now  is  worth  a  dollar  at  any  point 
in  the  future. 

Define  to  be  the  expecttd  reward  for  the  next  transition 
given  that  the  system  is  in  state  i.  v^(n)  is  defined  to  be  the 
expected  reward  of  operating  the  system  over  the  last  n  periods  (jumps) 
given  that  the  process  is  in  state jL,  n  periods  from  the  end.  v^(n) 
can  be  expressed  recursively  as: 

N 

v  (n)  =  q.  +  l  P  v  (n  -  i),  i  -  1,  ...,  N.  (C.l) 

1  -  j-1  X2  2 

N  is  the  number  of  states,  is  the  probability  of  a  transition  from 

state  i  to  j,  q  is  the  expected  rev  <  over  the  next  period,  and 
N  ± 

£  P  v  (n  -  1)  is  the  expected  reward  over  the  last  n  -  1  periods. 
j-1  12  2 

To  convert  this  process  to  an  infinite  horizon  process,  assume  that 

Ss  Y«>  *  <vi  +  ng>  .  (0.2) 

where  v^  is  the  reward  for  starting  in  state  i,  and  g  is  a  single 

reward  per  period  or  reward  rate  for  the  process.  Substitution  of 
Equation  (C.2)  into  (C.l)  yields 

N 

V.  +  g  -  q.  +  l  P..  v  ,  i  -  1,  ....  N.  (Co3) 

j-1  2  2 

This  system  of  N  equations  contains  N  +  1  unknowns  which  are  g  and  v^, 
for  i  *  1,  . . . ,  N. 

The  number  of  unknowns  in  Equation  (C.3)  needs  to  be  reduced  by 
one  if  the  system  is  to  be  solved.  Let  v^  *  +  z.  Substitution  into 

Equation  (C.3)  yields 
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a  a 

w  +z  +  g-q  +j;  P.,  w  +  * l  P  ,  i 

J-l  j  3  j-l  j 


1,  N. 

(C.4) 


Since  1  P  .  ■>  1,  Equation  (C.4)  becomes 

j-l  3 

»i  +  *  -  "i  +  I.1ptJwr 


(C.5) 


Since  Equation  (C.5)  is  of  the  same  form  as  Equation  (C.3),  the 
can  be  treated  as  relative  values.  This  allows  one  of  the  v^'s,  say 
v^,  to  be  set  to  zero,  thereby  reducing  the  number  of  variables  to  N. 

At  this  point,  it  is  convenient  to  develop  another  expression 
for  g,  the  gain  rate  of  the  system.  If  P  is  the  matrix  of  transition 
probabilities  and  0  is  the  vector  of  steady  state  probabilities  of 
occupying  the  states  of  the  system,  then. 


^  -  h  P 


(C.6) 


Multiplication  of  the  i  equation  of  Equation  (C.l)  by  0^  and 


summation  of  the  N  equations  yield 


N  N 


l  0±  g  +  l  vi  “  I  *i  +  l  l  Pi1  v  •  *>1 

i-1  1  i-1  1  1  i-1  1  1  i-1  j-l  13  •  1 


(C.7) 


However , 


N  N 


l  l  Pi4  ~  l  V4  l  ?i1  *  I  V4  ^  • 

i-1  j-l  J  1  j-l  3  i-1  13  j-l  3  3 


utilizing  Equation  (C.6).  Thus,  Equation  (C.7)  becomes 


l  0iqi 

i-1  1  1 


(C.8) 
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N 

since  ]>  0  -1. 

i-1  1 

The  development  thus  far  has  just  considered  the  evaluation  of 
the  gain  rate  of  the  process.  The  decisions  available  to  the  adminis¬ 
trator  have  not  been  considered.  If  k  -  1,  . ...  K  indexes  the 
alternatives  available  to  the  administrator  and  if  he  wants  to  maximize 
the  expected  reward  of  the  finite  horizon  problem.  Equation  (C.l) 
becomes 


vi(n) 


max 

k 


N 

{q  (k)  +  l  P^.Ck)  v.(n  -  1) ) 
1  j-1  2  2 


i-1,  ...,  N. 
(C.9) 


For  the  infinite  horizon  problem,  substitution  of  Equation  (C.2)  into 
Equation  (C.9)  yields 

N  N 

vi  +  ng  -  “J*  (q1(k)  +  |  ^  Pij<k)  Vj  +  (n  "  1'8  |  ^ijOO} 

N 

-  {q^k)  +  l  P^Ck)  Vj  +  (n  -  l)g>  , 

i-  1,  ....  N. 


This  can  be  written  as 

N 

g  ■  tqt(k)  +  |  ^  P^Ck)  vj  "  vi^  ’  1  “  1»  • 

v^  could  be  dropped  from  the  right-hand  side  of  Equation 


..,  N. 

(C.10) 

(C.10)  since 


it  is  unaffected  by  k. 


196 


C.2  Semi-Markov  Decision  Processes 

A  discrete  time  semi-Markov  process^  differs  from  a  Markov 

process  in  that  the  transitions  do  not  necessarily  take  place  at  evenly 

spaced  intervals  of  time.  Let  h^ (o)  be  the  holding  time  probability 

mass  function.  h.,(m)  is  the  probability  of  holding  m  periods  in 

00 

state  i  given  that  the  next  transition  is  to  state  j.  x  »  £  mh  (m) 

ij  m«i  iJ 

is  the  expected  holding  time  in  state  i  given  that  the  next  transition 
_  N  _ 

is  to  state  j.  T.  ■  I  P..  T  is  the  unconditional  expected  waiting 
i  j.!  *1 

time  in  state  1. 


For  a  finite  horizon,  discrete  time,  semi-Markov  process. 
Equation  (C.l)  becomes 

_  n  N 

v  (n)  «  q  T  +  l  l  P  v  (n  -  m)  h  (m),  i  -  1,  ...»  N. 

1  1  1  m-1  j-1  13  3  13 

(C.ll) 

As  n  becomes  large,  substitution  from  Equation  (C.2)  leads  to 

N 


r  +  ng  -  q  T  +  l  P ..  J  {v  +  (n  -  m)g)  h  (m) 
1  j-1  2  m-1  3  13 

N  __ 

ng  -  gx4.) 


Vi  +  ^ml  Pij  (vj  +  °*ij- 


N 


-  9^^  +  Z  1  Pij  vj  +  «8  -  gTj^  ,  i  -  1,  ....  N. 


Thus,  for  an  infinite  horizon,  Equation  (C.ll)  becomes 


N 


Vi  +  8Ti 


q  T  +  l  P  v  ,  i  -  1,  ....  N.  (C.12) 
1  1  j-1  13  3 


'‘’All  processes  considered  In  the  rest  of  this  appendix  will  be 
undiscounted  and  will  have  a  finite  state  space. 
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A  continuous  time  semi-Markov  process  allows  a  continuous 
distribution  of  holding  times.  For  a  finite  horizon,  Equation  (C.l) 


becomes 


vi(t)  -  q 


00 

t?i +  pij  |  VT>  V'  - T>dT  • 


i  -  1,  ....  N. 
(C. 13) 


Continuous  variables  t  and  x  are  used  to  index  time  and  q^  is  the 
expected  reword  per  unit  time  in  state  i.  x . .  becomes  an  integral, 


namely,  Jxh^TjdT,  while  x^  remains  £  x^  .  As  t  becomes 

large,  substitution  from  Equation  (C.2)  (with  t  replacing  n)  yields 

N 

vi  +  8t  •  Vi +  ^  Vtj  +  gt "  8V 

_  N 

“  qiTi  +  |  x  Pij  Vj  +  8t  "  gTi’  1  “  1*  “•*  N* 

Here,  g  is  the  gain  per  unit  time  rathe;  than  gain  per  period.  Thus, 
for  an  infinite  horizon.  Equation  (C.13)  becomes 
_  N 

Vi  +  8^i  "  qiTi  +  l  Pij  Vj  ’  1  "  1»  •••»  N*  (C.14) 

which  is  the  same  as  Equation  (C.12)  except  for  the  new  definitions  of 


terms. 


If  after  each  transition,  an  administrator  can  choose  an  action 


from  among  a  set  of  actions  indexed  by  k,  the  process  becomes  a 
continuous  time,  infinite  horizon,  semi-Markov  decision  process.  The 
action  chosen  affects  the  transition  probability  mass  function  and  the 
holding  t?m«.  density  function.  If  the  administrator  wants  to  maximize 
g,  the  expected  reward  rate  of  the  process,  Equation  (C.14)  becomes 
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g 


max 


N 


k  {qi<k)  +  l/Ti(k)}{  l  P^k)  vj  '  v^}  » 

1  «  1,  ....  N.  (C.15) 


Equation  (C.15)  suggests  the  following  algorithm  which  can  be  shown  to 
converge  to  the  optimal  solution  for  problems  like  those  of  this  paper: 


Algorithm  C,1  (Policy  Iteration  Algorithm) 

Step  1:  Set  v  ■  0  for  all  i.  d(0)  ■  0,  where  d(h) 
is  a  vector  of  actions  chosen  for  each  scate 
for  iteration  h  of  the  algorithm.  Set  h  ■  0. 
Step  2:  h  »  h  +  1.  For  each  state  i,  find  d^(h),  the 
index  of  the  action  that  yields 

_  N 

"£*(<^00  +  {i/Ti(k)>  {£  pij<k>  vj  “  vi^)  • 

Step  3:  If  d(ft)  ■  d(h  -  1) ,  stop. 

Step  A:  Solve 

v.  +  gTi{di(h)}  -  qi(di(h)}Ti{d1(h)} 

N 

+  l  P  {d  (h)}v  , 

j-1  X  J 

i  -  1,  ...,  N, 

for  g  and  v,  through  v„  , (v„  -  0) .  Go  to 
1  N-l  N 

Step  2. 

Algorithm  C.l  is  the  version  of  policy  iteration  used  in  this  paper. 


C.3  Example  Problem 

As  in  Section  B.A,  the  one  class  example  from  Tabl*  3.1  with 
X  ■  1  is  used  to  demonstrate  Algorithm  C.l.  The  remaining  parameters 
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* 


of  the  model  are  R  *  5,  C  ■  2,  and  y  «  3.  Since  n  ■  n  and 

o  s 

ng  *  [Ry/c]  *  7,  a  bound  on  the  size  of  the  state  space  is  seven. 

The  set  of  possible  actions  is  indexed  by  It  *  0  c r  1  (this  indexing 
scheme  agrees  with  the  body  of  the  paper  rather  than  with  Section  C.l). 
k  -  0  if  an  arrival  is  rejected,  and  k  -  1  if  an  arrival  is  accepted 
into  the  system.  To  avoid  a  trivial  system,  action  zero  cannot  be 
chosen  when  the  state  of  the  system  is  zero;  that  is,  if  the  system 
empties  out,  customers  must  be  allowed  back  into  it.  nQ  ■  7  implies 
that  action  one  cannot  be  chosen  when  the  state  of  the  system  is  seven. 
P(0)  is  the  matrix  of  transition  probabilities  for  alternative. 

k  -  0. 


To  State 

01234567 
P(0)  -  0-  -  -  --  --  - 

110000000 
201000000 
300100000 

From 

400010000 

State 

500001000 

600000100 

700000010 

The  dashes  in  a  row  indicate  that  the  action  cannot  be  selected  when 


the  system  is  in  the  state  represented  by  the  row.  For  alternative 
zero,  the  next  transition  is  sure  to  be  the  completion  of  a  service. 
?(1)  is  the  matrix  of  transition  probabilities  for  alternative  one. 


» 
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P(l) 


0 
1 
2 

3 

4 

From 
State  5 

6 

7 


0 

0 

0.75 

0 

0 

0 

0 

0 


1 

1 

0 

0.75 

0 

0 

A 

0 


To  State 
2  3 


0 

0.25 

0 

0.75 

0 

0 

0 


0 

0 

0.25 

0 

0.75 

0 

0 


4 

0 

0 

0 

0.25 

0 

0.75 


5 

0 

0 

0 

0 

0.25 

0 

0.75 


6 

0 

0 

0 

0 

0 

0.25 

0 


7 

0 

0 

0 

0 

0 

0 

0.25 


Pq^(1)  *  1  because  the  next  transition  is  sure  to  be  an  arrival  if 
alternative  one  is  chosen  when  the  state  of  the  system  is  zero.  For 
states  other  than  zero,  the  service  rate  competes  with  the  arrival  rate. 
Since  the  total  rate  out  of  a  state  other  than  zero  is  X  +  u  *  4  and 
since  U  provides  3/4  of  the  total  rate  and  X  provides  1/4  of  the  total, 
the  probability  that  the  next  transition  is  the  completion  of  a  service 
is  3/4  while  the  probability  that  the  next  transition  is  an  arrival  is 


1/4. 


T^(k)  is  simply  the  reciprocal  of  the  transition  rate  out  of 


state  i  under  action  k.  Thus, 

7(0)  • 


/  ’ 

and  t(1)  * 

1  ' 

0.33 

0.25 

0.33 

0.25 

0.33 

0.25 

0.33 

0.25 

0.33 

0.25 

0.33 

0.25 

0.33 

— 

. 
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Let  b(0)  be  the  matrix  of  expected  rewards  for  the  various 
transitions  under  alternative  zero.  b(0)  is  the  null  matric  because 
no  customers  are  allowed  to  join  the  system  under  action  zero.  b(l) 
is  the  matrix  of  expected  rewards  for  the  various  transitions  under 
alternative  one. 


b(l) 


0 
1 
2 
3 

From 
State  4 

5 

6 
7 


To  State 

0  1*  234567 

0  4.33  0  0  0  0  0  O' 

0  0  3.67  0  0  0  0  0 

0  0  0  3.00  0  0  0  0 

0  0  0  0  2.33  0  0  0 

0  0  0  0  0  1.67  0  0 

0  0  0  0  0  0  1.00  0 

0  0  0  0  0  0  0  0.33 


The  entries  b^  i+1(l)  are 


bi,i+l(1)  "  R~<i  +  3->C/y 


The  components  of  the  vector  of  expected  rewards  per  transition. 


ri(k),  are 


ri(k)  "  l  Pij(k)  bij(k) 


Here, 


r(0) 


and  r(l)  » 

r  4.333  ' 

0 

0.917 

0 

0.750 

0 

0.583 

0 

0.417 

0 

0.250 

0 

0.083 

. 0 

>  * 

The  expected  rewards  per  unit  time  in  state  i  under  action  k  are 
q1(k)  -  v^(k)/T^(k) 

Thus, 


M 

and  q(l)  - 

'  4.33  ' 

0 

3.67 

G 

3.00 

0 

2.33 

0 

1.67 

0 

1.00 

0 

0.33 

.  o 

«  . 

The  information  required  for  Algorithm  C.l  is  now  at  hand. 
Application  of  the  algorithm  proceeds  as  follows: 
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Step  Is  vQ  -  v1  -  . . .  -  0.  d(0)  «•  0. 

Step  2:  h  ■  1. 

d(l)  - 


Step  3:  Continue. 
Step  4:  g  ■  4.001. 

7.84 
7.50 

6.84 

5.85 
4.56 
3.00 
1.33 
0.00 

Step  2s  h  -  2. 


Seep  3s  Continue. 
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Step  4:  g  -  4.003. 

v  «  7.18  j 

6.85 
6.19 
5.23 
4.00 
2.67 
1.33 
0.00 

Step  2:  h  ■  3 

d(3)  -  1 

1 
1 
1 
1 
0 
0 
0 

. 

Step  3:  Stop. 

d(3)  indicates  that  n  ■  5  which  agrees  with  Table  3.1  as  does  the 

o 

optimal  value  of  g.  Section  4,6  contains  another  example  r f  policy 
iteration  and  gives  some  insight  into  the  relationship  between  d(l) 
and  the  optimal  policy  for  the  individual  optimum  problem. 


APPENDIX  D 


USER'S  GUIDE  AND  LISTING  OF  COMPUTER  PROGRAM 

Several  different  computer  programs  were  used  in  testing  the 
models  developed  in  Chapters  III  through  V.  The  program  that  is 
presented  here  was  used  for  many  of  the  tests  and  was  also  used  as  a 
starting  point  for  the  other  programs  that  were  developed ,  The  program 
is  written  in  FORTRAN  for  the  WATFIV  compiler  on  the  IBM  370-3033 
computer  at  The  Pennsylvania  State  University.  After  the  program 
and  its  use  are  described,  more  will  be  said  about  how  the  program 
was  adapted  for  other  uses. 

D.l  Introduction  to  the  Program 

The  program  is  called  POLI'?  to  reflect  the  fact  that  it 
performs  policy  iteration.  As  discussed  in  Appendix  C,  the  particular 
type  of  process  this  program  is  designed  to  solve  is  a  continuous  time, 
infinite  horizon,  finite  state  ^pace,  undiscounted,  semi-Markov 
decision  process.  This  progran  is  capable  of  solving  Models  I,  II, 
and  III.  However,  the  optimal  solution  for  Model  III,  the  Erlang 
service  time  model  is  the  phasa  o\  J  Aim  solution  which  cannot  always 
be  implemented. 

The  program  consists  of  two  parts.  The  first  builds  the  P(k) 

A 

matrices,  q(k)  vectors,  and  x(k)  vectors  from  the  input  data.  The 
second  part  performs  policy  iteration  to  solve  for  the  optimal  policy 
and  maximum  gain  rate.  Other  information  provided  for  the  optimal 
policy  is  the  set  of  v^'s  and  the  vector  of  steady  state  probabilities 
of  the  system  occupying  the  various  states,  0. 
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The  MAIN  routine  reads  the  input  data  and  prints  it  for  user 
verification.  MAIN  also  controls  the  flow  of  the  program.  The  sub¬ 
routines  are  described  in  the  order  in  which  they  are  called.  See 
Figure  D.l  for  a  flow  diagram  of  the  program. 

Subroutine  PROB  generates  a  transition  matrix,  P(k),  for  each 
—  M 

possible  action  k.  2  transition  matrices  are  generated  since  there 
M 

are  2n  possible  actions.  The  program  will  handle  up  to  five  classes 
(M  ■  5) .  Since  transitions  can  only  be  made  to  the  next  higher  or 
lower  state,  each  row  of  a  transition  matrix  contains  only  two  elements. 
The  first  element  is  the  probability  of  a  transition  to  the  next  higher 
state  while  the  second  is  the  probability  of  a  transition  to  the  next 
lower  state.  Subroutine  PR0B1  is  used  by  PROB  to  fill  in  the 
transition  matrices. 

Subroutine  PROFIT  generates  the  matrices  of  expected  rewards  of 

a  transition,  b(k) ,  for  each  possible  action  k.  Since  only  transitions 

to  the  next  higher  state  can  yield  a  nonzero  expected  reward,  each  row 

of  the  b(k)  matrices  contains  only  one  element.  Thus,  the  bi'k) 

matrices  are  dimensioned  and  used  as  vectors. 

Subroutine  EXPECT  first  generates  the  vectors  of  expected 

rewards  per  transition,  r(k),  from 
n* 

ri(k)  »  l  P  (k)  t  (k) 
j*o  2  2 

EXPECT  then  calculates  the  vectors  of  unconditional  expected  waiting 
times,  T(k).  Also,  using 

qt(k)  -  ri(k)/Ti(k)  , 

EXPECT  computes  the  vectors  of  expected  rewards  per  unit  time,  q(k). 


Figure  D.l  Flow  Diagram  of  POLIT 
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EXPECT  prints  the  code  which  relates  the  number  of  an  action  to  the 


classes  that  action  admits.  If  the  input  parameter  IFLAG  is  set  to 
one,  EXPECT  prints  the  condensed  matrices  P(k)  and  b(k)  along  with  the 
vectors  r(k)  and  q(k). 

Subroutine  POLIT  implements  the  policy  iteration  algorithm.  It 
utilizes  subroutine  SIMQ  from  IBM  (1970)  to  solve  the  system  of  linear 
equations  generated  by  Step  (A)  of  the  algorithm.  POLIT  outputs  g,  d, 
and  v  for  each  iteration  and  prints  the  vector  of  steady  state  proba¬ 
bilities,  0,  for  the  last  iteration  (which  yields  the  optimal  policy). 
In  addition,  POLIT  prints  the  vector  of  optimal  expected  rewards  per 
unit  time.  Finally,  POLIT  uses 


V  rtP*  P* 

8p*  -  I  0,  «t  . 

i-o 


where  P*  represents  the  optimal  policy,  to  again  compute  g.  The 
comparison  of  the  two  values  of  g  should  give  the  user  a  feel  for  the 
roundoff  error  that  exists  in  the  solution  of  his  problem. 


D.2  Use  of  the  Program 

The  program  input  is  described  first  and  then  illustrated  for 
the  example  described  in  Appendix  C.  Output  for  this  problem  is  given 
with  the  listing  of  the  program. 

Card  one  of  the  input  gives  the  number  of  classes  of  customers 
in  the  problem,  the  number  of  phases  of  service  (Erlang  number,  h) , 
and  mean  service  rate  for  each  phase  of  service.  The  user  is  reminded 
that  if  the  overall  service  rate  is  y,  then  the  service  rate  for  each 
phase  of  service  is  hy.  The  format  of  this  information  is  215,  F10.2. 
If  the  user  wants  to  see  the  P(k)  and  b(k)  matrices  and  the  r(k)  and 
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q(k)  vectors  that  are  generated,  he  should  also  Include  a  1  in  column 
25  of  this  card. 

Card  two  of  the  input  gives  the  reward  for  service  for  each 
class  of  customer.  The  reward  for  class  one  comes  first  followed  by 
the  reward  for  class  two  and  so  on.  The  format  for  the  rewards  is 
8F10.2. 


Card  three  of  the  input  lists  the  cost  per  unit  time  in  the 

system  for  each  class  of  customer.  Again,  the  costs  are  input  in  class 

order.  The  format  for  the  costs  is  also  8F10.2. 

Card  four  of  the  input  contains  the  self-optimum  balking  points, 

u  ,  for  each  class  m.  These  are  computed  from 
m 


»,  ■  [V/Cm+  (h  "  l)/2hJ 

a 

The  self-optimum  balking  points  are  input  in  class  order  using  a  format 

of  1615.  The  'lasses  must  be  numbered  so  that  the  class  with  the 

largest  n  is  class  one. 
s 


m 

Card  five  contains  the  arrival  rate  of  each  class  of  customers. 
The  arrival  rates  are  input  in  class  order  with  format  8F10.2. 

The  input  for  the  one  class  example  of  Appendix  C  follows: 

Card  One 

Column  5  10  11-20  25 

Input  1  1  3.0  1 

Card  Two 
Column 


1-10 


Input 


5.0 
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Card  Three 

Column 

1-10 

Input 

2.0 

Card  Four 

Column 

5 

Input 

7 

Card  Five 

Column 

1-10 

Input 

1.0 

The  output  for  this  example  Is  given  with  the  listing  of  the  program. 
Since  each  section  of  output  includes  descriptive  information,  the 
output  will  not  be  described  here. 

D.3  Other  Programs 

The  arrays  of  POLXT  had  to  be  enlarged  to  run  the  Erlang  service 
time  model  for  the  airport  example  in  Chapter  VI.  The  number  of  states 
required  for  the  Erlang  8  model  is  145,  so  the  dimensions  representing 
the  number  of  states  (50  in  the  listing)  must  be  at  least  145.  Also, 
since  the  optimal  policy  for  the  Erlang  model  was  not  implementable, 
POLIT  was  modified  to  check  policies  that  could  be  implemented  that 
were  near  the  phase  optimum  policy.  The  modifications  were  to  provide 
for  the  input  of  the  policies  to  be  tested  and  to  alter  subroutine  POLIT 
so  that  only  the  policy  evaluation  portion  of  Algorithm  C.l,  the 
determination  of  g  and  v,  was  performed  for  each  input  policy. 

Two  programs  that  are  on  the  IBM  370-3033  system  at  The 
Pennsylvania  State  University  were  also  used.  LPFREE  [see  Ilgen  (1978)] 
was  used  to  test  the  linear  programming  formulation  of  Models  I  and  II. 
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f 

f 

)' 

MPSX  [see  IBM  (1972)]  was  used  to  test  the  mixed  integer  programming 
formulation  of  Model  III. 

D.4  Listing  and  Output 

A  listing  of  POLIT  and  output  for  the  example  follow. 


.1 


I 


1 
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..  HAS?- II*** . 

.. ..START 

JOB 

1501....K3289646... 

. .  P.OS 

..  BMP  !!♦**.... 

. . . .START 

JOB 

1501....K3289646... 

..RUE 

..  HiS3-TT*** . 

....START 

JOB 

1501.. . .K 3289646... 

..ROE 

..  HAS^-II*1** . 

JOB 

1501.... K3289646... 

..POE 

..  HASP- II*** . 

.. ..START 

JOB 

1501....K32B9646... 

..PUE 

5:01  DATS:  06/12/7? 


HASP  SYSTEM  LOG 


JOB  «U3764,T*0029,R*0300C,S*230, 
tt  a  t ,  TR  AIN-T II ,  ?  0  8  as  *  1  6 
STS?  WAS  SXECU-ED  -  CON0  C JOE  OOCO 
SP  /SETTAr  /  STAP.T  79163.1115 
SP  /SETTAP  /  STOP  79163.1115  CPU 
CG 
* 


UT  00  * 

RXR03.DA"A1 
STEP  WAS  EXECUTED  - 
EP  /DATA  /  5T A ?T 

SP  /DATA  /  STOP 

OB  /K3289646/  3TAPT 
03  /K3289646/  STOP 


COtJD  CODE  0000 
79163.  11  15 
■>9163.1115  CPU 
79163.  11  IS 
79163.1115  CPU 


S'RUE  P  '  AA 

00000025 


OMIN  00.03SEC  MAIN  8K  LCS  OK 

00000050 


00011950 

0  MTV  00.S2SEC  MAIN  2P0K  LCS  OK 
OMIN  00.55SXC 


AT?  17  ***  19-3  (05/26/79—1049) 


1 


2 


C  SAIN  PROGRAM 

C***«*«* ************••**«*****. ************************* **»**•*•*«**•**« 

C  CLASS  ONE  SHOULD  RAPE '  LARGEST  P.SW ARD/C0S7,  CLASS  2  SETT  LARGEST 

C  AND  SO  CM. 

C  DEE INTTICNS 

C  I EL AG  -  IT.SE.  1,  SHORT  O/P;  IP  1,  O/P  INCLUDES  P,3,R,C  MATRICES 

C  LCt ASS  -  NUMBER  C?  CLASSES  OF  CUSTOMERS 

C  SEWARD (L)  -  CLASS  REWARDS  TOP  SEE  VICE 

C  COS"  (L)  -  CLASS  TIME  COSTS 

C  SLIM  (L)  -  ?0 RCED  CLASS  BALKING  POINTS  ••  CCSTOMIES  III  SYSTEM 

C  AS  0?  19JAN79  ONIY  KLla(l)  IS  USED 

C  G  -  MEAN  SERVICE  RATE 

C  KPHASE  -  NUMBER  CP  PRASES  OE  SERVICE 

C  PLAMEA(L)  -  CLASS  ARRIVAL  PATES  GIVES  STATE  <  tJLIM(L) 

C  I  -  ST A"E  0?  SYSTEM  -  NUMBER  OP  PHASES  OF  SERVICE  IS  THE  SYSTEM 

C  J  -  1  IP  STATE  BECOMES  I  ♦  FPRASE  APTEP  TFANS,  2  IF  1-1 

C  P(t,I,J)  -  CONDENSED  TRANSITION  MATRICES 

C  S  (L,  I)  -  EXPECTED  PROFIT  /  OCCUPANCY 

C  Q(L,I)  -  EXPECTED  FPORIT  /  UNIT  TIME 

C  T<L,I)  -  EXPECTED  WAITING  TIME  IS  STATE 

:  v (ij  -  relative  value  c?  starting  in  s^ate  i 

C  G  -  GAIN  RATE  -  EXPECTED  PROFIT  /  TIMS 

C  ID  (I)  -  POLICY  CHOSEN  THIS  ITERATION 

C  B  (L,  I)  -  VECTORS  OP  EXPECTED  PPOfIT  OP  TPANS  FROM  I  -  I*KPHASE 

£*•«**•••******«*••*»*•••***••***•*****«•*••••***•*«**•***•*.*•**•«*•«*• 
COMMON  LCLASS,  REWARD  (  50),  COST  (  50),  NHM(  50),  9,  KPHASE, 

1  FLAMD  A  (  50),  P(  32,  50,  2),  R(  32,  50  )  ,  Q  (  32,  50),  T(  32,  50), 

2  V(  50),  G ,  ID (  50),  B(  32,  50),  SSTATS 

<;**•*  **************************************  >* **************** *********** 
C  READ  INPUT 

C********************************** ************************** *********** 

IN  -  5 


tut 
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10  fit  -  6 

READ  (XU,  100)  LCLASS,  KFH ASE,  0,  ITLAG 
100  PORHAT  (2X5,  no.  2,  15) 

READ  (IS ,200)  (REWARD  (1) ,  L  -  1,  LCLASS) 

200  PORHAT  (DUO.  2) 

READ  (IS,  200)  (COST(L),  L  •  1,  LCLASS) 

P*AC (IS,  300)  (SLXS(L),  L  -  1,  LCLASS) 

300  POSH AT (1615) 

R3AC(I!t,  200)  (PLAHCA(l),  L-  1,  LCLASS) 

C*»-*n  ••* ************  ******  ********  ********** a***************  *********** 

C  ECHCCHECX  OP  I SPOT 


12 

V» ITS  (TOUT, 

400) 

13 

400 

FORMAT  f  1H  1) 

14 

WHITE (IOUT, 

500)  LCLASS,  K PRASE,  U 

15 

500 

POP H AT ( 1H0 , 

10X,  'THIS  QUEUE  CONTROL  PRCBLEfl  HAS', 15,  •  CLASSES  O' 

1,  «P  COSTOHSSS.  THS  MSA8  SERVICE  PATS  OF*//20X, •  THE  ERLAXG',IS, 

2  »  S2R7SR  IS  •,  P10. 2) 

WRITE(IOOT,  600) 

600  POP  MAT (1  HO ,  10X,  • IMP  OT  VALUES  OP  THS  P ARAHSTSBS  FOR  BACH  CLASS', 
1  '  ARB  GIVES  IS  THE  FOLLOWING  TABLE') 

WRITS (IOUT ,  ’00) 

700  PORMAT ( 1H0 ,  13X.  'CLASS',  10X,  • REWARD* ,  15X,  'COST',  16X, 

1  'SLIM',  IPX,  ' PLANCA') 

DO  5010  L  *  1,  LCLASS 

5010  WRITE  (IOU",  300)  L,  P.SWAPD(l),  COST  (L)  ,  NT.IM  (L)  ,  PL  A  HD  A  (L) 

300  FORMAT  (180 ,  2X,I5,  3X,  P10.2,  10X,  F10.2,  12X,  IS,  13X,  P10.2) 
NSTATS  -  1  +  <P  RASE  *  NLIM  (1) 

;*****•■ ****** «**•******•*»••**•* ***•*•**••  *******  ***********  **••*•***< 

:  G  SW  PR  AT  S  TRANSITION  MATRICES 


CAM.  PRO'v 

>****«**t>«** a************* ****************************** i 

CESSU.'TE  EXPECTED  PROFIT  OP  REXT  TRANSITION  MATRICES 

**********  .>****  *  ***********************************  *****  ■ 

CALL  PR02/L? 

t?  (ITLAG. NS. 1)  GO  TC  5047 

.**«**« •**«*»«*•  *****  *******  ****************************  i 

OUTPUT  P  AS a  B 


27 

LCLP1  -  2  ** 

LCLASS 

23 

DO  5045  L  ■ 

1,  LCIP1 

29 

WHITE (IOUT, 

2100)  L 

30 

2100 

FORMAT  ( 1H0  , 

■>0X,  'P  AND  3  .MATRICES  FOLLOW  POP  AC7TCN',IS) 

31 

WRITE (IOUT, 

2200) 

32 

2200 

FORMAT  ('HO, 

3 OX ,  ' TRANSITION  PROBABILITIES') 

33 

DO  50  30  I  « 

1,  ttSTATE 

34 

5030 

WRITS  (IOUT  , 

2300)  (P (1,1, J)  ,0-1,2) 

35 

2300 

FORHAT  (8 F10. 

6) 

36 

WRIT* (IOUT, 

2400) 

37 

2*00 

FORMAT  (  1HC  , 

30X ,  'EXPECTED  REWARDS  POP  TRANSITION') 

3® 

WR ITS (TOUT, 

2300)  (B  (1,1)  ,1-1,  US  TATE) 

39 

504  5 

CONTINUE 

GEM  SR ATS  EXPECTED  PROPIT/OCCUPAKCT  AND  PXPSCTED  RSVARD/TIME 


5047  CAT!  EXPECT 

IP  (TPLAG.NE. 1)  GO  TC  5C60 


OUTPUT  R  AND  Q 


70  5050  L  «  1 ,  LCLP1 
vo  its  (tout,  250  0)  L 

2500  FORMAT  (1R0,  23<,  *R  AND  Q  VECTORS  FOLLOW  FOE  ACTION',  15) 
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i»5 

WRITS (IOOT, 

2630) 

46 

2600 

FORMAT ( 1H0 , 

301, 

•R  (I)  •) 

47 

BRITS (IOOT, 

2700) 

CH  (1,1)  , 

I  ■  1,  NSTATI) 

48 

2700 

FORMAT  (8F10. 

49 

WRITE (IOOT , 

:»3C0) 

50 

?,800 

FORMAT  ( 1  HO  , 

30X, 

•Q(I)  M 

51 

MR  ITT (IOOT, 

2700) 

(C  (L,I)  , 

I  »  1,  ESTATE) 

52 

5050 

CONTINUE 

£****•*««*«««*••«*•****•*••*•***• 

C  PESFORM  POLICE  ITERATTOR 

C««M«***««M**M*i«»*M<M«W*** 

53  5060  CALL  POtl? 

5U  STOP 

55  310 


56  SUBROUTINE  PR08 

57  COMMON  LCLASS,  SERAHDf  50),  C0ST(  50),  SLIM  (  50),  0,  Km  AS  2, 

1  rLANDA(  50),  P(  32,  50,  2),  R(  32,  50),  0  (  32,  50),  T{  32,  50), 

2  7  (  50)  ,  G,  ID  (  50),  0<  32.  50),  51 STATS 

C*** ******** *********************** ********** ******************* ******* 
C  2  *•  L CLASS  TRANSITION  MATRICES  WILL  SB  GENERATED 

C  1ST  -  ICLASS  CLASSES  TAKES  0  AT  A  TIME 

C  2S0  -  LCLASS  CLASSES  TAKES  1  AT  A  TINS 

C 
C 

C  LAST  -  LCLASS  CLASSES  TAKES  LCLASS  AT  A  TIME 

C  SOTS  -  SO  PAR  5  IS  UPPER  LIMIT  OS  LCLASS 

C**' 

53 

59 

60 
61 

62  10 

C**' 

c 

<:** 

63 

64 

65  20 

C«*' 

C 

c 

c** 

66  LL  ■  1 

6 7  DO  100  L  «  1,  LCLASS 

63  TLAMOA  ■  FLANOA(L) 

69  LL  *  LL  ♦  1 

70  100  CALL  PROU1 (IL,?LANDA) 

C**** *** *********** ••••»*****•**  **•**•»•••*•*•**•**•***••*****•-< 

C  GENERATE  TRANSITION  MATRICES  PON  LCLASS  CLASSES  TAKEN  2 

C  AT  A  TIN! 

(;«•*•*■•**•«-««*«*»«*•«***»**«••••***•««■*•*••**•***»*•****•«••• 

71  I* (LCLASS. LT. 2)  GO  -O  1O00 

72  LCLN1  «  LCT.ASS  -  1 

73  DO  20 n  L  •  1,  LCL.H 

74  L?1  ■  L  ♦  1 

75  DO  200  LI  »  L? 1 ,  LCLASS 

76  LL  «  LL  ♦  1 

77  TL1MDA  *  FLA3DA  (L)  ♦  FLAMDA  (L  1) 

79  CALL  PROB1 (LL,  TL1NEA) 

79  200  CONTINUE 

C*««t*«**M«*****MM**««*««**«MM>M*M**«n*«**»»M«**tM>M 

C  GENERATE  TRANSITION  MATF ICES  POR  LCLASS  CLASSES  TAKEN  3 

C  AT  A  TINE 

<;*••«•*«••*•••••***•*•*•••*••**  -*  ••*•*•*•**••* 


LCLP1  ■  2  **  LCLASS 
DO  10  L  ■  1,  LCLP1 
00  10  I  -  1,  ESTATE 
00  10  J  «  1,  2 
P(L,I,J)  ■  0.3 

*************** ******** ««*****+*************+***»* ******** ******** 

GENERATE  TRANSITION  MATRIX  POR  LCLASS  CLASSES  TAKEN  0  AT  A  TINE 


DO  20  I  -  2,  ESTATE 
J  -  2 

P(1,I,J)  ■  1.0 


GENERATE  TRANSITION  NATSICSS  POR  LCLASS  CLASSES  TAKEN  1 
AT  A  TIMS 
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I?  (IC1ASS.lt. 3)  30  TO  1000 
LCLM2  ■  LOLA? 3  -  2 
00  300  I.  •  1,  LCLS2 
L?1  ■  L  «•  1 

00  300  Li  ■  LP1,  LCLM1 
Li  PI  «  11  ♦  1 
DO  300  L2  »  L  IP  1 ,  L CLASS 
LL  •  LL  ♦  t 

TLAflOA  •  fLAHOA  (L)  4  FLAMDA  (L  1)  *  FLAMDA (L2) 
CALL  PR0S1 (LL,  TLAMCA) 


300  continue 

C««*****««*****««««*******«****«**« **«*«*******«*««*•**«•%•**•**•*«**•*» 

C  GENERATE  TRANSITION  MATRICES  FOP  LCLASS  CLASSES  TAKEN  '4 

C  AT  A  TIMS 

C ******* •**«****#*« ************ w **«***»« *****•«***»«•****•*•*  *********** 

IP  (LCLASS.LT. 4)  00  TO  1000 
LCLM3  ■  LCLASS  -  3 
DO  400  L  •  1,  LCL.13 
LP1  *  L  ♦  1 

00  400  LI  «  LP1,  LCLS2 

LIP  1  «  LI  *■  1 

00  400  L2  -  L  IP  1 ,  LCLMl 

L2P 1  «  L2  ♦  1 

00  400  L3  -  12P1,  LCLASS 

LL  »  LL  ♦  1 

-LAMDA  ■  PLAMOA(L)  4  FLAHOA  (LI)  ♦  PLAMCA(L2)  ♦  FLAHOA  (L31 
CALL  PROBl (LL,  TLAMCA) 

400  CONTINUE 

C******** ••****••* *.*******•*•****«********•**•********•*****••*.*«****• 

C  SEN  SPATE  *f8ANS  ITICN  MATRICES  FOR  LCLASS  CLASSES  TAKEN  5 

C  AT  A  TIME.  NOTE  FOB  THIS  PROOF AM,  SAX  VALUE  OP  LCLASS  IS  5. 

£**************************  ********  **************************  *********** 

IF  (LCLASS. LT. 3)  30  *n  1000 
TLAMCA  -  0.0 
DO  500  L  ■  1,  5 

5C0  TLAMOA  «  TLAMOA  ♦  FLAHCA(L) 

LL  ■  LL  +  1 

CALL  PR081  (LL,  TLAMCA) 

TOUT  ■  6 

IF (LCLASS.  3T. 3)  SPITS (IOUT,  2000) 

2000  FORMAT ( 1H0  ,  20X,  ‘PROBLEM  HAS  MOPS  THAN  5  CLASSES.  ONLY  CLASSES', 

1  '  TO  5  SILL  RE  USED. •) 

1000  RE TURK 
END 

SUBROUTINE  PFOB 1 ( LL ,  TLAMCA) 

COMMON  LCLASS,  RESARD (  50),  COST(  50),  NLIM (  50),  0,  KPHASE, 

1  FLAMOi (  50),  P(  32,  50,  2) ,  R  (  32,  50),  0 (  32,  50),  T(  32,  50), 

2  V(  50),  G,  ID(  50),  E(  32,  50),  1ST  A"1! 

C*n**<***M**«*«*t*<M**»***H**t***t*«M**t*M***«**««M****««*t»i>«**« 

C  PR0B1  FILLS  IN  9  MATRIX  FOR  SOEPCHTINB  PPOB 


NSM1  ■  tt  STATE  -  1 
TOTAL  «  TLAMCA  ♦  U 
PUP  •  TLAMCA  /  TOTAL 
PUOSN  -  U  /  TOTAL 
9  (LL,  1,  1)  ■  1.0 
DO  40  I  ■  2,  SSM1 
?  (LL,  r,  1)  *  PUP 
P (LL,  r,  2)  «  PUOSN 
RETURN 
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127  SUBROUTINE  PROFIT 

128  cohson  lclass,  rewabd(  50} ,  cos?(  50) ,  slih(  so> ,  a,  kpha se, 

1  PL AS C A (  50),  ?  (  12,  50,  2) ,  H  {  32,  50),  Q{  32r  50),  ?(  32,  50), 

2  7(  50),  0,  ID  (50)  ,  3  (  32,  50),  N3TATE 

£******«*****•*•****•*•**••■%*•«•**•**•**•***•* ••*«*********••*,*  ******* 

C  2  **  LCLASS  EXPEC-ED  PROFIT  OP  THE  NEXT  TRANSITION  RATRICES 

C  WILL  81  GENERATE!. 

C  1ST  -  LCLASS  CLASSES  TAKES  0  A?  A  TI3E 

C  2SD  -  LCUSS  CLASSES  TAKES  1  AT  A  TIRE 

C 
C 

C  LAS?  -  LCLASS  CLASSES  TAKES  LCLASS  AT  A  TIRE 

£*•****••******«••* **C* *•*****•*«•«* ****** ****** ******* *********  ******* 

129  NSM1  »  ESTATE  -  KPHASE 

110  LCLP1  «  2  **  LCLASS 

131  DO  10  L  •  1,  LCLP1 

132  DO  10  I  -  1,  3S TATS 

133  10  B(L,  I)  -  0.0 

C»**«a«*»*«««*k****M*»««M*«*t*«***M***t*******M**tMt*»*»»*«M*iM 

C  POE  LCLASS  CLASSES  TAKER  0  A?  A  TIRE,  3-0. 

C 

C  GENERATE  PRO  PIT  HATP.IX  POP.  LCLASS  CLASSES  TAKEN  1  AT  A  TIRE. 

£*•**••*••***«•*****•*•**•,**.*****•**.**•.***•.•.••*•**•*..»*•*,,,*•.• 

124  LL  -  1 

135  DO  100  L  -  1,  LCLASS 

136  LL  ■  LL  ♦  1 

137  T1ATE  -  PLARCA(L) 

138  DO  50  I  •  1,  NSS1 

119  J  ■  I  ♦  KPHAS! 

140  PJ  -  J  -  1 

141  SOI  -  PLAROA(L)  *  (”2 HARD  (L)  -  *J  *  COST  (L)  /  0) 


142  50  3  (LL ,  I)  •  SHE  /  -RATE 

143  100  CONTINUE 

<:•*••**«•«*••••—•*****«*•**••*••••*•*••••**•***••*•**•«••***•:!****-•* 

C  GENERATE  P30PI”  RATRICES  FOR  LCLASS  CLASSES  TAKER  2  A”  A  TIRE 

O—  •  •*  *  ********** **•••*«*•*** *****************  ************  **•*•«**•« 

144  17 (LCLASS. LT. 2)  GOTO  1000 

145  LCL R 1  ■  LCLAS3  -  1 

146  CO  200  L  -  1,  LCLR1 

147  LP1  ■  L  ♦  1 

1*8  DO  200  LI  -  L? 1 ,  LCLASS 

140  LL  -  LL  *  1 

150  TEAT*  •  KLAHEA  ( L)  *■  FLARCA  (Li) 

151  DO  150  I  *  1,  NSR 1 

152  J  ■  I  *  FPHASJ 

153  K.T  -  J  -  1 

154  SON  •  FURDAU)  *  ( REWARD (L)  -  RJ  •  COST  (L)  /  0)  * 

1  FLAKUML1)  •  (REWARD  (LI)  -  ?J  *  COST(LI)  /  0) 

153  150  3  (LL,  I)  -  S"'1  /  TRA^T 

156  200  CORTIin* 

£ **•••«***•****  **«*  ********41  ******* *••*•**•*•*****•****• *****  •  >••••**•*: 

C  GENERATE  "10  FT”  TA'RTCES  FOP  LCTAFS  CLASSES  TAKER  2  AT  A  TIRE 

£***««•**«**•*****•»••••**»*•*•**•*******•••***••*•***.******•***••*•* 

15"  I?  (LCLASS,  LT.  3)  GO  TO  1010 

158  LC1R2  «  LCLASS  -  2 

159  DO  300  L  -  1,  LCL 12 

160  LP1  -  L  ♦  1 

161  DO  300  LI  ■  L?1 ,  ^CIRI 

162  L 1 ? 1  -  LI  *  1 

163  DO  300  L2  -  T.1P1,  LCLASS 

164  LL  -  LL  *  1 

165  7"  ATE  ■  FLARCA  (L)  ♦  FLARE  A  (LI )  *■  FLARCA  (L2) 

186  DO  250  I  -  1,  3SR  1 
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167 

J  ■  I  *•  KPHASt 

168 

FJ  •  J  -  1 

169 

SOS  ■  FLABDA  (L)  *  (REWARD  (L) 

-  FJ  *  COST  (L) 

/ 

0) 

1  ♦  FLABDA (LI)  *  (REWARD  (LI)  - 

FJ  *  COST  (LI) 

/ 

0) 

2  *  FLABDA (L2)  *  (BI»APD{L2)  - 

FJ  *  COST  (L2) 

/ 

0) 

170 

250 

8 (LL.  I)  ■  SOB  /  TSATE 

171 

300 

COSTXNOS 

c* 


172 

173 

174 

175 

176 

177 
173 

m 

190 

191 
182 

183 

184 
135 
186 


C  GE5 S31TE  PROFIT  HATRICFS  FOB  LCLASS  CLASSES  TAKER  4  AT  A  TISE 

(;«***•*»•*****«**•*** •*••*»**** ********«******«**********••****«**«***, 

IF  (LCLASS.LT.  4)  GO  TO  1000 
LCLN3  •  LCt AS 3  -  3 
00  400  t  «  1,  LCL2I3 
LSI  -  l  ♦  1 

DO  40C  LI  ■  L?1 .  1C1B2 

LIP  1  •  \M  ♦  1 

DO  400  L2  -  L IP  1  *  LCLB1 

L2P1  •  12  «•  1 

DO  400  L3  »  L2P1 ,  LC1ASS 

LL  »  LL  ♦  1 

TRATS  «  PLABDA(L)  ♦  riABDA(LI)  ♦  FLAME  A  (L2)  *  FLAHDA  (L3) 

DO  350  I  “  1 ,  HSnl 
J  ■  I  ♦  KPHASE 
FJ  «  J  -  1 

SOB  ■  FLABDA  (l)  *  (FEWARO  (L)  -  FJ  *  COST  (L)  /  0) 

1  FLAB  DA  (LI)  *  (  REWARD  (Li)  -  FJ  *  COST  (LI)  /  0) 

2  4  FLABDA (L2)  *  (BEWA8D(L2)  -  FJ  *  COST  (L2)  /  7) 

3  *  FLABDA  (L3)  *  (REGARD  (L3)  -  FJ  *  COST  (L3)  /  IJ) 


187 

350 

B  (LL.  T)  ■  SOB  /  TSATE 

188 

400 

COUTISOE 

c 

GEMERATE  PROFIT  MATRIX  FOP  5  CLASSES  TAKEN  5  AT  A  TIBS 

c 

POT?  UPPER  BOnBD  OS  LCLASS  IS  5. 

£**«* 

199 

I?  (LCLASS.LT.  3)  GO  TO  1C00 

190 

LL  *  LL  ♦  1 

191 

TSATE  «  0.0 

192 

DO  430  L  ■  1,  5 

193 

430 

TEATS  ■  TRATE  +  FLABDA  (L) 

194 

DO  500  t  -  1,  USB  1 

195 

J  «  I  ♦  KPHA53 

196 

FJ  -  J  -  1 

197 

SOB  -  0.0 

198 

DO  450  L  »  1,  5 

199 

450 

SOB  ■  SOB  *  FLABDA ( L)  *  (REWARD  (L)  -  FJ  *  COST  (L)  /  U) 

200 

500 

8  (LL.  I)  ■  SOB  /  TSATE 

201 

1000 

RSTORH 

202 

SOD 

203 

S09EC0TISE  FTPECT 

204 

COB BOW  LCTASS,  REWARD  (  50),  COST  (  50),  WLIB  (  50),  0,  KPKAS3, 

1  FLABDA  {  50),  P(  32,  SO,  2),  R<  32,  50),  Q(  32,  50),  T(  32, 

2  7  (  50),  G,  ID  (  52),  B(  32,  50),  H  ST  ATE 

50)  , 

c 

R  (  ,D  -  B(  ,1)  •  ?(  ,1.  1) 

N 

£***« 

205 

LCLP1  ■  2  <"*  LCLASS 

208 

9571  ■  US'1' ATT  -  EPHA57 

207 

• 

DO  10  1  »  1,  LCtS 1 

20  8 

DO  10  I  •  1,  KSTATS 

20^ 

m (L  ,1)  «  0.0 

210 

Q  f  L ,  I)  ■  0.0 

21 1 

10 

’(L,I)  •  0.0 

1 


218 


212  DO  50  t  ■  2,  LCLP1- 

213  DO  40  I.-  1,  nSfll 

214  40  J(U)  -  P(t,I,  1)  *  3(1,1) 

215  SO  CONTINUE 

C  **•****«****«****••***«**•**•••***••«***•*•••**■,***•**».*, *••*****«« 

C  GENERATE  +  t OS  LCLASS  CLASSES  TAKES  0  AT  A  TIME 

C**********-.**.************..*****.*  *******«•**••**•*****,*.*.. *•**,., 

216  IOUT  -  6 

217  LI  ■  1 

213  l  »  0 

219  WRITE  (lOU*,  20)0)  IL,  L 

220  2000  PORRAT  (1H0 ,20.<,  'ACTIOS  ’  ,15  ,  '  ADMITS  CLASSES',  SIS) 

221  DO  60  I  ■  2,  NSTAT2 

222  60  T(LL,  I)  ■  1/0 

C*«<k«*»********** ••«•*•*•** ******** ************************** ******** 

C  GENERATE  *  TOR  LCLASS  CLASSES  TAKES  1  AT  A  TIRE 

C ******************************************************* ************* 


223 

DO  100  L  ■  1,  LCLASS 

224 

LL  ■  LL  +  1 

225 

TEATS  «  EL  AREA  (L) 

226 

8RITE  (IOUT ,  2000)  LL,  L 

227 

T  (LL ,  1)  ■  l/TRATE 

223 

TRATZ  »  TEATS  *  0 

229 

DO  70  I  -  2,  MS1 1 

230 

70 

T  (LL ,  I)  *  l/tSATS 

231 

100 

CONTINUE 

c«*** 

|l********** 

c 

GENERATE  T  FOR  LCLASS  CLASSES 

TAKEN  2  AT  A  TIME 

c**** 

||***  ******  ********  *****  1 

232 

I? (LCLASS. LT, 2)  GO  TO  1000 

233 

LCLM1  *  LCLASS  -  1 

2  34 

DO  200  L  -  1,  LCLR1 

233 

LP1  -  L  ♦  1 

. 

236 

DO  200  LI  ■  LSI ,  LCLASS 

237 

LL  *  LL  ♦  1 

• 

238 

"RATE  »  ELAMOA(L)  ♦  PLAMDA(LI) 

239 

WR  ITS  (TOOT,  200  0)  LL,  L,  LI 

24  0 

T  (LL,  1)  *  1/  TRATE 

241 

TR  ATE  «  TRATE  *  tt 

242 

DO  1^0  1*2,  NSE1 

243 

170 

T  (LL,  t)  •  1/  TRATE 

244 

200 

c  **** 

c 

CONTINUE 

GENERATE  *  FOR  LCLASS  CLASSES 

"AKEN  3  AT  A  TI8E 

24S 

IP (LCLASS.LT. 3)  GOTO  1000 

246 

LCLM2  •  LCLASS  -  2 

247 

DO  300  L  ■  1,  LCLM2 

243 

LP1  «  L  ♦  1 

249 

DO  300  LI  »  LSI,  LCLH1 

2S0 

LI  PI  *  LI  ♦  1 

251 

DO  300  12  ■  L  IP  1 ,  LCLASS 

252 

LL  *  LL  *  1 

25  3 

T?.  ATE  ■  PLARCA(L)  *  PLAMDA  (LI)  ♦ 

PLAREA iL2) 

254 

WRITS  (IOIJT ,  2000)  LL,  t,  LI,  L2 

255 

T  (LL,  1)  ■  1/  TRATE 

256 

TR  ATE  ■  TRATE  ♦  0 

257 

OO  270  1*2,  HSR1 

253 

270 

T (LL,  I)  «  1/  TEATS 

25  0 

300 

CONTINUE 

C  GENERATE  ”  *0?  LCLASS  CLASSES  TAKEN  4  AT  A  TIEr 

C  *******  *•*****•*<•*•*••  *************************  ******** 

260  I*  (LCLASS.LT. 4)  GO" T0  1000 


v 


261 

LCLM3  ■  L CL ASS  -  3 

262 

DO  400  L  *  1,  LCLM3 

263 

L?  1  ■  L  ♦  1 

264 

DO  400  LI  -  LP1,  LCLM2 

265 

L1P1  ■  Li  ♦  1 

266 

00  400  L2  *  11  PI ,  LCL.11 

267 

L2P1  ■  L2  ♦  1 

269 

DO  400  L3  -  L2P1,  LCLASS 

269 

LL  «  LL  *  1 

270 

TRITE  «  PLABBA(L)  *  PL  ABE  A  (LI ' 

271 

SR  ITS ( lOHT ,  2000)  LL,  L,  LI,  1 

272 

T (LL, 1)  •  1/TRAT! 

273 

TRATS  -  TP  ATI  ♦  0 

274 

DO  370  1*2,  BSH1 

275 

370 

T (LL,  I)  *  1/  TEATS 

276 

400 

COSTTRHS 

C 

GERSRATE  T  FOR  5  CLASSES  T 

C 

ROTS  LCLASS  IS  BCOHDSD  BY 

277 

I? (LCLASS.LT. 5)  GO  TO  1000 

273 

LL  •  LL  ♦  1 

279 

TRATS  -  0.0 

290 

DO  430  L  «  1,  5 

281 

430 

TRATS  -  SPATE  *  FLAMCA (L) 

292 

WRITE  (10 U"\  2000)  LL,  (I,  L  * 

29  3 

T  (LL,  11  -  1/  TRATS 

284 

SPATE  *  TRATS  +  0 

295 

DO  470  1  *  2,  HSS 1 

296 

470 

I  (LL,  I)  «  1/  TP  ATS 

297 

1000 

COtlTIHOS 

C*«*F 

1,  5) 


C  G3MSSATE  a  POP  Alt  POLICIES 

C  ********  ************  ****** *********************  **********  *  *********** 


293 

DO  1090  1*2, 

299 

o 

O' 

o 

0(1, 1)  -  P(1,D 

290 

DO  1200  L  •  2, 

291 

no  1100  1*1, 

29  2 

1100 

Q  (L » I)  *  P  (L,  I) 

293 

1200 

CORTIRIJE 

294 

RSTUFH 

295 

SOTO 

296 

S73FC0TIRE  PCLI' 

2Q7 

COH BOH  LCLASS, 

IIS  SI 


(1,1) 


299 


29° 

io’o 

301 


c< 

c 

c 

c 

c 

c 

c 

c« 

c< 

c 

c 

c* 


1  Ft,  ABO  A  (  50),  ?(  32,  50,  l'*,  R<  32,  50),  0  (  32,  50),  T(  32,  "0)  , 

2  't  (  50),  o,  ro(  so),  e(  32,  50)  ,  estate 

I  •*• *********  **************** ***•*****•*•«*••**• ********* ****•••*•*, 

USE  POLICY  S**SA,i,TC3  TO  FTRD  OPTIMAL  POLICY  -  HR  DISCO  TRTSr , 
ITiriST-?  HORIICR. 

AA  9 ILL  COHTAtH  CO°P?ICTSir,S  0?  SYSTEM  CP  LINEAR  ECTTATIOMS  III 
V  ( 1 )  ,  ,  V(SSTAT">1)  ,G.  ROTE  V  (RSTATE)  *  0.0  DOT  DILL  CO  STAIR 

0  FOR  CCR73RISMCE. 

DR  SILL  BE  RKS  OF  SYSTEM. 

t ******************************** a* ******  ****************  *** * • ****** 

DI3SM5IOM  AA  (2500)  ,  EB(501,  I  DOLE  (  50) 

»***•*«***•***• «*•*«*••***** ******************************** *•***«*■ 

DI.MERSIOKS  0*  A  A  A  ME  BB  MOST  PS  EXACTLY  !I  PI  M  AMD  R  POR  SI  MO 
THESE  ARE  AUTOMATICALLY  TAKES  CARE  OF  BELCH 

■  **«  ************* ************** *****************************  *»**•**« 

its a  «  o 

DO  10  I  ■  1,  RSTATE 

Y(t)  *  0.0 


SI 


*>  ),X*  u  A'w  t  -  ■  ’ 


n  ci  n  n  n 


10  ioold(I)  *  o 

LCLP1  -  2  •+  LCLASS 
SS3 1  x  ESTATE  -  1 
10 07  -  6 

is  itep.  »  ms  ♦  i 

00  1000  1  -  1,  HSTATP 

PSAX  -  0,0 

ICHOIC  -  0 

DO  100  L  -  1,  LCLP1 

1311  -  I  -  1 

IP1  -  I  +  7PHAS! 

I?  (I. £0.1)  SOS  «  P(t,I,1)  *  V  (IP  1) 

I?(I.GT.  (PSTATE-KPHASE)  )  GO  TC  18 

17(1. MS. 1)  SOS  *  P(1,I,1)  *  7  (IP  1)  *  ?  (L,I,2)  •  V(XR1) 

GO  TO  19 

18  SOS  -  ?(t,T,2>  *  7  (TS1) 

19  17  (T(L,I)  .EQ.0.0)  C,C  TO  100 

TOT  At  -  Q  ( L  ,  I)  *  (1/T.(t,I))*  (SOS  -  V  (I) ) 

C*******  **;*****  ************  ********  ************************************* 

AO SIT  AS  SAflY  CLASSES  AS  POSSIBLE  IP  A  TIE 


IF  (TOTAL. G2.F3AX)  SO  TC  60 
SO  TO  100 
60  73 AX  -  TO-AL 

LCHOIC  -  L 
100  CONTI3UJE 

TP  (LCHOIC.  HP.  ))  GO  TO  110 
WHITE  (IOUT,  2000) 

2000  FOPS  AT (1H0, 201,  'LCHOIC  «  0») 
STOP 

110  ID (I)  -  LCHOIC 


SET  OP  A  A  IK  83 

********************  **•***<!*  1>4*«**  ***************** 

DO  120  J  «  i,:jshi 

IXJ  -  I  *  (S-1)  *  USt ATE 

AA(IXJ)  *0.0 

IP  (J.EQ.  (I-D)  AA(I3U)  *  - P  (ECHOIC,  I,  2) 

120  I?(J.  ”0.  (I  +  7P3ASE)  3  AA(XXJ)  =»  -P'LCHOIC,  1 ,  1) 

IXI  «  I  ♦  (I  -  1)  *  HSTATS 


AA(XXI)  *  1.0 

I  lit  -  I  *  (HSTATE  -  1)  *  3ISTATE 
AA(IXtl)  *  T  (ICHOIC, I) 

33(1)  *  S  (LCHOIC,  I) 

1000  COHTtN’JE 

CALL  S I3Q ( Al, B8 , HSTATE ,KS) 

IP(KS.SQ.O)  GO  TO  200 
WPTTEdO’iT,  210  0) 

2100  FOSS  AT  ( 1H  0  ,  20X,  'SINGULAR  SYSTEM') 

STOP 

200  WHITE  (IOUT,  2200)  IIEP. 

220C  *0  RS AT ( 1  HO ,  20X,  ' ?  FSPLT  3  FOR  ITERATION  15,  '  FOLLOW ') 
WHITE (IOUT ,  2300) 

2300  FOR  H  AT  (1  HO ,  23X,  "dCISTCM  VECTOR ' ) 

WRITE  ( 10 OT ,  2400)  (ID  (I)  ,  I  3  1 ,  HSTATE) 

2400  70 PS AT (1615) 

DO  1100  I  -  1,  MSTAT'' 

1100  7(1)  ■  33  (I) 

S  *  7  (ESTATE) 

7 (MSTA"E)  "  0.0 
WRITS 'TOUT,  2500)  S 

2500  FOSSA?  (1HO,  21X,  '  G  A 1 3'  3  ',  F10.3) 

WRITE  (ICnT ,  2500) 

2600  FOSSA- (1H0,  20X,  '7  TCT CP*) 


221 


360 

UP.  ITS  (10 a-,  2700)  (V(I),  1*1,  MS?  ATE) 

361 

2700 

FCRSATfBFI?.  4) 

362 

DO  1200  1*1,  NSTATE 

363 

1200 

17 (12 (I). HE. ISOLD (I) )  GO  TO  1500 

364 

GO  TO  1700 

36S 

1530 

COMTIMUB 

366 

DO  1600  1*1,  MSTATE 

36  7 

1600 

IDOLB(I)  *  ID(I) 

368 

GO  TO  15 

369 

1700 

7FXTE(TCUT.  2300) 

370 

2900 

FORK AT ( 1H0 ,  23X,  'POLICE  ITERATION  TERMINATED' ) 

C**  tg«4 

c 

CORPBTE  STEADY  STATE  PMOBA BILITIES 

£****' 

371 

00  3200  1*1,  ESTATE 

372 

LCHOIC  »  ID  (I) 

373 

00  3120  J  -  1,  SSM1 

374 

IXO  *  (1-1) *NST ATE  ♦  J 

375 

AA(IXJ)  *  0.0 

376 

17  (J.EQ.  (1-1) )  AA(IXJ)  *  P  (LCHOIC,  I,  2) 

377 

3120 

r»(J.FQ.  (I*K?HASE))  AA  (170)  -  P  (LCHCIC,  I,  1) 

378 

I XI  *  I  ♦  (1-1)  •  MSTATE 

379 

AA  (III)  *  AA(IXI)  -  1.0 

380 

IXM  *  t  *  MSTATE 

381 

Ai(IXS)  *1.0 

3?2 

B8  (I)  0.  0 

383 

3200 

COSTIBOE 

384 

3B(  MSTATE)  *1.0 

385 

CALL  SIHQ(AA,BB, MSTATE, MS) 

386 

IF ( KS. EQ.  0)  GO  TO  3  3CO 

387 

WRITE  (IOIJT ,2100) 

388 

STOP 

389 

3300 

TOIV  *  0.0 

390 

OO  3340  1*1,  MSTATE 

391 

LCHOIC  *  10(1) 

34  2 

7(1)  -  Q  (LCHOIC ,  I) 

393 

38(1)  *  3B(I)  *  T  (LCHOIC, I) 

3340 

TO  17  ■  XDIV  +  DC  (I) 

3«5 

XXULT  *  0.0 

396 

00  3360  1*1,  MSTATE 

397 

B8  (I)  »  BB  fl)  /  '<017 

39? 

3360 

txult  *  xnutr  *■  bb(I)  *  7  (i) 

399 

URITE  (lOtJT ,  3400) 

400 

340G 

FORK  AT  (1H0,  2CX,  '  STEADY  STATE  PROBABILITIES  ') 

401 

99 1TB (IOUT, 2700)  (B 3  ( I) ,  1*1,  MSTATE) 

402 

WRITE  (TOUT,.!  6  30) 

40  3 

360  0 

FOPRAT ( 1 HO ,201 , 'OPTICAL  Q  VALUES '  5 

404 

9?  ITT  (IOO'r, 2700)  (7(1),  1*1,  MSTATE) 

405 

7PrrE(T07~,3730) 

40  6 

37n  C 

FOBS A?  <  H  2CX,  '  GAIH  RITE  CHECK') 

407 

3RITr  (IOU".  i^OO)  XBOLT 

408 

27TUP3 

409 

"no 

410 

S7BF0nmIME  SIHQ  (A,B,7,KS) 

41  1 

DIKEMSIO  1  A(1),  B  (1 ) 

C*< 

C 

PURPOSE 

C 

OB? AIM  SOLUTION  OF  A  SE*  OF  SIMULTANEOUS  LIMEAF.  ECATIOMS  , 

c 

AX*8 

c 

USAGE 

c 

CAM  STHQ  (A,8,M,  K:') 

C 

C7SCFIP-I0M  0?  ?  Ar  A  SET  EPS 

222 


C  1  -  MATRIX  0?  CO EFT ,  STORED  COLUMNWISE.  THESE  ABE 

C  DESTROYED  IS  COMPUTATION.  SIZE  GP  A  ROST  BE  EXACTLY  M  BY  H 

C  E-  VECTOH  0/  ORIGINAL  BBS  VALUES.  THESE  APE  REPLACED  BY  SOLS. 

C  LENGTH  OT  3  MOST  BE  EXACTLY  S. 

C  S  -  NUMBER  OP  IQUATIOSS  AND  7 ABIABLES.  S.GT.ONE. 

C  PS  -  OUTPUT  DIGIT 

C  0  FOR  NORMAL  SCLUTICN 

C  1  YOR  SINGULAR  SET  OR  EQUATIONS 

C  METHOD 

C  METHOD  OT  SOL'JTIOS  IS  BY  ELIMINATION  USISG  LARGEST  PI70TAL 

C  DIYTSOR.  EACH  STAGS  0?  ELIHISATICS  COSSISTS  OF  INTERCHANGING 

C  POWS  WHSH  NECESSARY  TO  AVOID  PITXSIOH  BY  ZERO  OE  SHALL 

C  ELEMENTS . 

C  THE  FORWARD  SCIOTICJJ  TO  OBTAIN  7ASIABLE  !t  IS  DCNE  IB 

C  N  STAGES.  THE  BACR  SOLUTICN  FOR  THE  OTHEB  VARIABLES  IS 

C  CALCULATED  BY  SUCCESSIVE  SUBSTITUTIONS.  PINAL  SOLUTION 

C  VALUES  A3E  DEVELOPED  IS  V2CT0R  B,  WITH  VARIABLE  1  IN  3(1),  ETC. 

C  IF  MO  PIVOT  CAS  BE  FOUND  EXCEEDING  A  TOLEPANCE  OF  0.0, 

C  THE  MATRIX  IS  CONSIDERED  SINGULAR  AND  KS  IS  SET  TO  1.  THIS 

C  TOLERANCE  CAN  3E  MODIFIED  BY  REPLACING  THF  FIRST  STATEHEST. 

<;*«•*****•****«***•«•***•*»•**••***•••*******»«* •«********«** **»**•**««• 
C 


426 
-*  27 

i*2? 


C< 

c 


c**« 

c 

c *** 

35 

c 

c  **« 


FCRV19D  SOLUTION 


c***< 

412 

TOL  «  0.0 

413 

K7  «  0 

414 

.IJ  «  -S 

4  15 

DO  65  J  «  1,  M 

416 

JY  -  J  ♦  1 

417 

JJ  ■  JJ  ♦  N  ♦  1 

418 

BIG  A  »  0 

410 

IT  ■  JJ  -  J 

420 

DO  30  I  «  J,  N 

c***1 

C 

SEARCH  FOR  MAXIMUM  COEFFICIENT 

c  +**' 

421 

IJ  »  IT  *1 

422 

IT  (ABS  (BIG A) - ABS  (\  (IJ) ) )  20,30,3  0 

423 

20 

BIG  A*  A  ( IJ) 

424 

IMAX  «  I 

425 

30 

CD NT TNUE 

TEST  FOR  PITOT  LETS  THAN  TOLEFANCF  (SINGULAR  MATRIX) 

,«»*•***•****»«* •  **•**»**»*••***•••,<»*«*•** *•*•*•*•.•*»* 

17  (ABS  (BIGA)-TOL)  35,  35,  40 
*1  ■  1 
RETURN 


IN'ERCHANG" 

■  »*■*****«»*< 


SO '.IB  17  NECESSARY 


429 

40 

ii  »  j  ♦  ::  *  (j  -  2) 

430 

I"  »  TMAX  -  J 

431 

DO  50  X  *  J,  :l 

432 

I1»  11  ♦  N 

433 

12  «  11  *  IT 

434 

SAVE  «  A  (11) 

435 

A  (11)  *  A  (12) 

436 

A  (12)  -  SAVE 

C 

DIVIDE  EQUATION  BY 

C***1 

43'r 

f  - 

A  (1 1 )  »  A  (T  1 )  /  3IG  A 

43R 

SA7Y  »  B(IMAT) 

223 


439  C  (IS  AX)  »  3(J) 

440  3 ( J)  •  SAFE/3XSA 

C«a«*  •—***+•»«*•*•** *n— •«»-•-***«  — ••*•*••*•*«•••*»••••**** »«**«•—** 

C  ELIflIXATS  BZXT  VARIABLE 

« a****** 


441 

I?  1  J-*0  53,  70,  5 

442 

S5 

ICS  «  N  •  (J-1) 

44  3 

90  65  U  -  JY,  N 

444 

IX J  *  IQS  ♦  IX 

44  5 

I?  -  J  -  IT 

446 

99  60  JX  -  JY,  N 

447 

IXJX  -  a  • ( IT- 1) 

44  3 

JJX  -  IXJX  ♦  IT 

449 

<0 

A (IXJX)  -  A  (IXJX) 

450 

65 

C-— • 

3  (IX)  -  B  (XX)  -  ( 

♦  IT 

-  {»(ixj>*KJjin 
(3(J)  •  k  (IXJ) ) 

r  «••«»-*««  or*  **»*i 


c 

C— •»  — 


BACK  SOLOTCOH 


451 

70 

NY  •  N  -  1 

452 

IT  -  N  •  N 

453 

DO  90  J  ■  1,  SIT 

454 

n  -  rr-j 

455 

la  «  n  -  j 

456 

IC  «  5 

457 

DO  90  5  •  1,  J 

458 

B (IB)  •  3 (IB)  - 

459 

IA  •  IA  -  N 

460 

80 

IC  -  IC  -  1 

461 

SBTOHS 

462 

• 

IN  3 

STATEMENTS  EXECUTE C«  5782 


A(IA1 


*  3(2C) 


COPE  0SAG7 
DIAGNOSTICS 
,C0T»TL5  TIME- 


OBJECT  CODE-  2*098  3TT25, AFSAT  AREA-  50020  BETTS, TOTAL  ANSA  A  7  AILABLE- 
308933  C T  5H8Q33-  ') ,  N0HBE3  OF  NABNINGS-  P,  NUMBST  CF  EXTfNSK 

0.37  SEC , 3 XECOTTOU  TIM-  0.09  SEC, 


•"’ilS  QUEOE  COtTSOl  PROBLEM  .-US  1  CLASSES  OF  CDSTOREPS.  THE  RE  AN  SEP7ICT  SATE  OF 
THE  ERLANG  1  S  2H7F-.  IS  3.00 

IMPOT  TALOES  OF  THE  PARAMETERS  FOR  EACH  CLASS  APE  GIVEN  IS  THE  POLIOVINO  TABLE 


CLASS 

HZ1ARD 

COST 

NLIB 

rLAT?  A 

1 

5.70 

2.00 

7 

1.'" 

K 


0.3 

0.0 

C.O 

0.0 

0.0 

0.0 

0.0 

0.3 


P  AND  8  MATRICES  FOLLOV  FOB  ACTION  1 


0.0 

1.000000 
i.ooe,'<*o 
1 .000000 
1.000000 
. .000000 
1.  V50OC0 
1.000000 


TRANSITION  P 7 03 ABILITIES 


224 


V 


r 

ir 


stsio’ed  hivahds  fob  transition 


0.0 

0.0  0.0  0.0  0.0  0.0  0.0 

P  AND  8  RATRIC2S  fOtLOB  FOR  ACTION  2 

TRANSITION  P80I 1 BIX IT I IS 

0.0 

1.000000 

0.0 

0. 250000 

0.750000 

0.250000 

0.750000 

0.250000 

0.750000 

0.250000 

0.750000 

0.250000 

0.750000 

0.250000 

0.750000 

0.0 

0.0 

sx?2ct*o  asxiaos  rap  transition 

4.333333 

1.666667  3  .000000  2.333334  1.666667  1  .000000  0.33.1334 

ACTION  1  ACHII5  CIA S SIS  0 

ACTION  2  ADRIT3  CLASSSS  1 

8  AND  Q  VECTORS  rOXLOV  FOR  ACTION  1 

8(1) 

0.0 

0.0 

0.0  0.  0  0.0  0.0  0.0  o.c 

Q(X) 

0.0 

0.0 

0.0  0.0  0.0  0.0  0.0  0.0 

P  AND  Q  VICTORS  FOLLOW  FC3  ACTION  2 

0.0 

F(U 

4.331 

C.917  0,750  0.513  0.417  0.250  0.0*1 

0(11 

o.o 

4.33.1 

3.667  3.000  2.333  1.667  1.000  0.331. 

RESULTS  FOR  IT ISA tION  1  FOLLOW 

DECISION  VECTOR 

0.0 

2  2 

2  2  2  2  2  1 

GAIN  -  4,001 

7  VECTOR 

7.s»1*»1 

7.5046  6.1416  5.3535  4.5561  3.0001  1.3316 

RESULTS  FOR  r'lRAriON  2  FOLLOW 

DECISION  VICTOR 

0.0 

2  2 

2  2  2  1  1  1 

GAIN  «  4.003 

V  VICTOR 

1767 

6.5461  6.1705  5.2252  4.0027  2.6696  1 . 3  U  2 

3ISULTS  FOP  IT!" AT ION  3  FOLLOW 

DECISION  VECTOR 

0.0 

2  2 

2  2  2  1  1  1 

i,i 

i 


4 

< 

i 


\ 


3113  « 


4.003 


r  vector 


7. 17fi7 

6.3461 

6.1905  5.2262 

4.0027 

2.6635 

1.3342 

0.0 

POLICE  ITERATION  IBRHX8 ATZD 

0.6676 

0.2225 

STEADY  STATE  PROBABILITIES 
0.3742  0.0247  3.0032 

0.0027 

-0.0000 

-0.0000 

4.3333 

3.6667 

OPTISAL  Q  VALUES 
3.0000  2.  333  3 

1.6667 

0.0 

o 

• 

o 

0.0 

4.0027 


8118  8 ATI  CHECK 


ACCOUNT:  U37«!» 

OATt:  08/2<»/79  IDE NT: 

USE?:  RUE  ? 

0ESTINATIC8:  AA 

OS-21.3  HASP-2. T5G  370/3033 


NAXIflUa  TT'15  {SEC):  20  NS"  CPU  (SEC):  1 

ACTUAL  TINS'.  INCLUDING  2."  SEC  STSTE*!  TINT:  r  *  '.  17 

LINES  PaiirSD:  7.30  CAnDS  PUNCHED:  0  3  *.35 

NAXINUJ1  3ECPROS:  5000  TO"M  .’’SCOPUS:  700  3  <.12 

CARES  HEAD:  712  *••**  TOTAL 


VITA 


The  author  was  born  in  Grafton,  North  Dakota,  on  April  5,  1947. 

He  attended  primary  school  in  Cavalier,  North  Dakota,  and  Menomonie, 
Wisconsin,  where  he  also  attended  secondary  school.  He  entered  the 
United  States  Air  Force  Academy  in  1965  and  received  the  B.  S.  degree 
in  Mathematics  in  June  1969. 

The  author  continued  his  education  at  the  University  of  Florida 
where  he  received  the  M.  S.  degree  in  Systems  Engineering  in  March  1975. 

He  is  a  member  of  Alpha  Pi  Mu,  Phi  Kappa  Phi,  Pi  Mu  Epsilon,  and 


Sigma  Xi. 


